Tuesday, January 13, 2015

Data masking with dbms_redact

dbms_redact basicly is a 12c package which was backported to 11.2.0.4. This package allow us create very customizable masking procedures. For simplicity, let say you need to mask credit card numbers, cutting off the middle 6 digits in 16-digital cards for some user i_see only


Create test env
 SQL> create user i_see identified by "***";  
 User created.  
 SQL> grant create session,create table to i_see;  
 Grant succeeded.  
 SQL> grant execute on dbms_random to i_see;  
 Grant succeeded.  
 SQL> conn i_see/***  
 Connected.  
 SQL> set num 20  
 SQL> create table mask_tab (card_num varchar2(20));  
 Table created.  
 SQL> insert into mask_tab  
 select (round(dbms_random.value(1000000000000000, 9999999999999999)))   
 from dual connect by level <= 10;  
 10 rows created.  
 SQL> commit;  
 Commit complete.  
 SQL> select * from mask_tab;  
 CARD_NUM  
 --------------------  
 4315180567707642  
 8343744789583160  
 7219646392845285  
 1281700031661329  
 6939207428729261  
 2520681905985755  
 7344297730838169  
 9104025795743973  
 4902051003122352  
 2026891028511588  
 10 rows selected.  



Lets create redact policy for user i_see, table mask_tab and column card_num


 [testhost:oracle@testdb]$ sqlplus "/as sysdba"  
 SQL*Plus: Release 11.2.0.4.0 Production on Tue Jan 13 12:31:44 2015  
 Copyright (c) 1982, 2013, Oracle. All rights reserved.  
 Connected to:  
 Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production  
 With the Partitioning, OLAP, Data Mining and Real Application Testing options  
 SQL> begin  
  dbms_redact.add_policy(object_schema     => 'I_SEE',  
              object_name      => 'MASK_TAB',  
              column_name      => 'CARD_NUM',  
              policy_name      => 'CARD_NUM_POLICY',  
              function_type     => DBMS_REDACT.REGEXP,  
              expression      => 'SYS_CONTEXT(''USERENV'',''SESSION_USER'')=''I_SEE''',  
              regexp_pattern    => '(\d{6})\d{6}(\d+)',  
              regexp_replace_string => '\1******\2');  
 end;  
  11 /  
 PL/SQL procedure successfully completed.  
 SQL> select user from dual;  
 USER  
 ------------------------------  
 SYS  
 SQL> select * from i_see.mask_tab;  
 CARD_NUM  
 --------------------------------------------------------------------------------  
 4315180567707642  
 8343744789583160  
 7219646392845285  
 1281700031661329  
 6939207428729261  
 2520681905985755  
 7344297730838169  
 9104025795743973  
 4902051003122352  
 2026891028511588  
 10 rows selected.  
 SQL> conn i_see/***  
 Connected.  
 SQL> select * from mask_tab;  
 CARD_NUM  
 --------------------------------------------------------------------------------  
 431518******7642  
 834374******3160  
 721964******5285  
 128170******1329  
 693920******9261  
 252068******5755  
 734429******8169  
 910402******3973  
 490205******2352  
 202689******1588  
 10 rows selected.  
 SQL> exit  
 Then drop policy   
 [testhost:oracle@testdb]$ sqlplus "/as sysdba"  
 SQL> begin  
     dbms_redact.drop_policy(  
      object_schema => 'I_SEE',  
      object_name  => 'MASK_TAB',  
      policy_name  => 'CARD_NUM_POLICY');  
  end;  
  7 /  
 PL/SQL procedure successfully completed.  
 SQL> conn i_see/***  
 Connected.  
 SQL> select * from mask_tab;  
 CARD_NUM  
 --------------------  
 4315180567707642  
 8343744789583160  
 7219646392845285  
 1281700031661329  
 6939207428729261  
 2520681905985755  
 7344297730838169  
 9104025795743973  
 4902051003122352  
 2026891028511588  
 10 rows selected.  


Also, keep in mind if user has  EXEMPT REDACTION POLICY  privilege, then policies dont applies to user (DBA role has this priv)


Wednesday, November 19, 2014

One more password gen function

I'd like to share function(one of the many=) to generate passwords in PL\SQL. You can easy Google a lot of pretty smart examples, but some time ago I was looking for example and couldn't find password generators with requirements:
1. To be random "enough" 
2. Password must consist of Upper,Lower case and digit.  
Second requirement was strict, even tho it reduces randomnicity of result.
Here my example:

create or replace  function gen_passwd(v_num in varchar2) return varchar2 is  
   v_ret   varchar2(100);  
   v_str   varchar2(100);  
   v_salt  varchar2(100);  
   -- If password contains "@" - connect to Oracle DB with given password drops with ORA-12154  
   -- Excluded "@" from val_spec array  
   val_spec varchar2(100) := q'[!#$%&()*+/<=>?\{}[]]';  
   val_num  varchar2(100) := q'[0123456789]';  
   val_upper varchar2(100) := q'[ABCDEFGHIJKLMNOPQRSTUVWXYZ]';  
   val_lower varchar2(100) := q'[abcdefghijklmnopqrstuvwxyz]';  
  begin  
   if v_num > 30 then  
    raise_application_error(-20008,  
                'Maximum password length in Oracle is 30 characters!');  
   else  
    if v_num < 7 then  
     raise_application_error(-20009,  
                 'Minimum password length must be 7 characters!');  
    end if;  
   end if;  
   v_str := val_num || val_upper || val_lower || val_spec;  
   ---genarate v_num random(ok,pseudo) numbers from array  
   for i in 1 .. v_num loop  
    v_ret := v_ret ||  
         substr(v_str, trunc(dbms_random.value(1, length(v_str))), 1);  
   end loop;  
   --genarate 3 values according to requirements (1 from every array)  
   v_salt := substr(val_num,  
            trunc(dbms_random.value(1, length(val_num))),  
            1) ||  
        substr(val_upper,  
            trunc(dbms_random.value(1, length(val_upper))),  
            1) ||  
        substr(val_lower,  
            trunc(dbms_random.value(1, length(val_lower))),  
            1);  
   ---replace 3 values starting from random position between 1 and v_num-3  
   v_ret := replace(v_ret,  
            substr(v_ret,  
               trunc(dbms_random.value(1, length(v_ret) - 3)),  
               3),  
            v_salt);  
   return v_ret;  
  end;  
 Here we go  
 SQL> declare  
  2  v_out varchar2(30);  
  3 begin  
  4  for i in 7 .. 30 loop  
  5   select gen_passwd(i) into v_out from dual;  
  6   dbms_output.put_line(v_out);  
  7  end loop;  
  8 end;  
  9 /  
 R10BqH<  
 e2Hlsh\C  
 m8CtHODok  
 7Isl(&Ji1v  
 B>!xS4PaUcV  
 f&/3L1YfW#OE  
 5Pi<}Wi[B#CMV  
 Z54Nofc?J{+P3I  
 f3jzrn4P1HrAcJU  
 AkV40Rm#gsTa#ZDU  
 p2wp4IhAD1zT9>o6&  
 [f8IfUCDxy%QYU6z0E  
 pm8Ml>fsTs1dy68#9>T  
 Z+5KnBwe?\}4x2j5f(yK  
 GtUAazUlfh$lzq2n5EhI#  
 tof<}Qpk7Mu<88ae#petls  
 S3vdEU0pSxuJ7*y76Mf6&av  
 Cdf96Yn%%\Ql)q%SiTiwEbwz  
 (Mjpr<AfO8Pu4FS?QkdpF1a>\  
 5SuHLad*a87GeXni<sf4HW*[0D  
 A}dHgu}i>qa9Y><Ky=Ba04Lla6g  
 n(\Q8Ew}hHT#\}Aro=Bn#YICk41v  
 zm#$JK/$}N>W}PtPTHCN2DhG/{3s[  
 CmFI1bmG62zx=b\7Ln/*[Urk&dId39  
 PL/SQL procedure successfully completed  

Friday, April 5, 2013

ORION 11.2.0.3 - Oracle I/O Numbers


11g database has included officialy unsupported tool for storage IO measurement. Can be found in $ORACLE_HOME/bin, release of which is similar to DB release. There is almost no official information about Orion 11.2.0.3 neither MOS or documentation. What is the point to use this tool? Well, first of all is - Orion makes IO workload similar to database workload, i.e can make OLTP, DSS ( OLAP\DWH systems) or mixed workload with the same block size as database load. Another reason is - although tool
is unsupported - it's developed by Oracle from release to release, so I think it will be ever supported. It can run in 5 modes:


  • oltp - loads small IOs only ( 2-32K, default 8kb)
  • dss - loads large IOs only (>=128kb, default 1m)
  • normal - variable combination of 8kb and 1m workload
  • simple - small IOs workload, then large IOs workload 
  • advanced - complex workload based on election matrix, which defining mix and interferention of small and large IOs.

Be carefully, by default orion just READ data, but with "-write" key it's going to WRITE data to mount points and ERASE all existing data. Be carefully(!). I'm going to read only. So, we need to create file with list of mount points which we want to test:

[oracle@test.local orion]$ mount |awk '{print $1}'|grep /dev/mapper > mytest.lun

[oracle@test.local orion]$ cat mytest.lun
/dev/mapper/dl-sdb01
/dev/mapper/dl-sdb02
[oracle@test.local orion]$ sudo ./orion -run oltp -testname mytest -num_disks 24 -duration 30
All time in microseconds(us) =1/1000000 sec. Unfortunatly there is an issue to test in NORMAL or ADVANCED mode. Key "-duration", does work only in SIMPLE,OLTP and DSS modes,so. Default duration is 60seconds per "one point sample", so in ADVANCED mode - its about 24(oltp)*24(dss)*60sec= ~9hours.I dont't have so much time, so   I tested OLTP (24*30=~12minutes)
The output log would be:

ORION VERSION 11.2.0.3.0
Command line:
-run oltp -testname mytest -num_disks 24 -duration 30 
These options enable these settings:
Test: mytest
Small IO size: 8 KB
Large IO size: 1024 KB
IO types: small random IOs, large random IOs
Sequential stream pattern: one LUN per stream 
Writes: 0%
Cache size: not specified
Duration for each data point: 30 seconds
Small Columns:,     24,     48,     72,     96,    120,    144,    168,    192,    216,    240,    264,    288,    312,    336,    360,    384,    408,    432,    456,    480
Large Columns:,      0
Total Data Points: 44
Name: /dev/mapper/dl-sdb01 Size: 1247999950848
Name: /dev/mapper/dl-sdb02 Size: 1498416087040
2 files found.
Maximum Small IOPS=5321 @ Small=480 and Large=0
Small Read Latency: avg=90199 us, min=40826 us, max=599406 us, std dev=29438 us @ Small=480 and Large=0
Minimum Small Latency=7640.00 usecs @ Small=24 and Large=0
Small Read Latency: avg=7640 us, min=114 us, max=110447 us, std dev=4834 us @ Small=24 and Large=0
Small Read / Write Latency Histogram @ Small=480 and Large=0
Latency: # of IOs (read) # of IOs (write) 
        0 - 1 us: 0 0
        2 - 4 us: 0 0
        4 - 8 us: 0 0
        8 - 16 us: 0 0
       16 - 32 us: 0 0
       32 - 64 us: 0 0
       64 - 128 us: 43 0
      128 - 256 us: 48 0
      256 - 512 us: 13 0
      512 - 1024 us: 5 0
     1024 - 2048 us: 1419 0
     2048 - 4096 us: 28472 0
     4096 - 8192 us: 101814 0
     8192 - 16384 us: 46385 0
    16384 - 32768 us: 9416 0
    32768 - 65536 us: 709 0
    65536 - 131072 us: 12 0
   131072 - 262144 us: 0 0
   262144 - 524288 us: 0 0
   524288 - 1048576 us: 0 0
  1048576 - 2097152 us: 0 0
  2097152 - 4194304 us: 0 0
  4194304 - 8388608 us: 0 0
  8388608 - 16777216 us: 0 0
 16777216 - 33554432 us: 0 0
 33554432 - 67108864 us: 0 0
 67108864 - 134217728 us: 0 0
134217728 - 268435456 us: 0 0


Saturday, December 29, 2012

Oratop-light,simple DB monitoring tool for Linux

This tool probably useful for system administrators, or like synopsis overview for DBA. This Oracle tool is really easy-to-use, very lightweight and simple for cursory Database performance overview. It can be downloaded from MOS [ID 1500864.1]. This tool made for Linux(x32 and x64) and 11gR2 only. Just run it authorizing with some privileged user:
[oracle@dbtest oratop]$ ./oratop 




oratop - utility for near real-time monitoring of databases, RAC and Single Instance [ID 1500864.1]

Active Data Guard SLA pursuit

When your business users runs read only report from ADG (physical standby database opened READ ONLY) there may unacceptable lag between primary and standby databases.Since 11.2 there is a nice view to see this lag.

SQL> SELECT * FROM V$STANDBY_EVENT_HISTOGRAM WHERE NAME = 'apply lag' AND COUNT > 0 and last_time_updated like '%12/29/2012%';

NAME         TIME UNIT          COUNT LAST_TIME_UPDATED
------------ ---- ---------- -------- --------------------
apply lag       0 seconds     3780291 12/29/2012 09:54:04
apply lag       1 seconds      319929 12/29/2012 09:52:32
apply lag       2 seconds        6509 12/29/2012 07:10:20
apply lag       3 seconds        3868 12/29/2012 05:52:06
apply lag       4 seconds        2956 12/29/2012 05:52:07
apply lag       5 seconds        2223 12/29/2012 05:46:56
apply lag       6 seconds        1344 12/29/2012 05:46:57
apply lag       7 seconds        1118 12/29/2012 05:46:58
apply lag       8 seconds        1144 12/29/2012 05:33:56
apply lag       9 seconds        1094 12/29/2012 05:33:57
apply lag      10 seconds        1098 12/29/2012 05:33:58
apply lag      11 seconds        1001 12/29/2012 05:33:59
apply lag       2 minutes        6363 12/29/2012 02:34:41
apply lag       3 minutes        1786 12/29/2012 02:35:54

61 rows selected

So, in my environment 2-3 minutes lag - unacceptable. Of course the best way - is tuning apply speed, but it's not always possible, for network limits for example. But we may guarantee users some SLA, via this kind of trigger,violation of which will lead to "ORA-03172: STANDBY_MAX_DATA_DELAY of 60 seconds exceeded". Run this trigger on primary, then it will by applied on standby. Here is simple Oracle example:

SQL> CREATE OR REPLACE TRIGGER LOGON_TO_STANDBY_TRIGGER
AFTER LOGON ON some.schema
BEGIN
IF (SYS_CONTEXT('USERENV','DATABASE_ROLE')
IN ('PHYSICAL STANDBY'))
THEN execute immediate 'ALTER SESSION SET STANDBY_MAX_DATA_DELAY=60;'
END IF;
END;



Friday, December 28, 2012

Golden Gate plugin for EM Cloud Control 12c

Today I'd like to tell how to monitor Golden Gate infrastructure in EM Cloud Control 12c.
Requirements:
- Oracle GoldenGate release 11.2.1.0.1 or later ( I used  version 11.2.1.0.3)
- Oracle EM Cloud Control 12c Bundle Patch 1 (12.1.0.1) or later ( I used  Bundle Patch 2, i.e. 12.1.0.2)
- Any release of JDK 1.6 (not JRE). I used latest for today 1.6.0_38. Check java version:

[oracle@dbtest ~]$ $JAVA_HOME/bin/java -version
java version "1.6.0_38"
Java(TM) SE Runtime Environment (build 1.6.0_38-b05)
Java HotSpot(TM) 64-Bit Server VM (build 20.13-b02, mixed mode)

- Any OS where GG can run except : HP NonStop, IBM iSeries, and IBM z/OS. I used OEL Linux 5.6
- I assume EM CC12c and agents 12c are deployed on all required targets.

Steps:
- Configure plug-in and java
- Download plugin via EM CC12c. Setup->Extensibility->Plug-ins->Middleware->Oracle GoldenGate 
- Deploy plugin on Management Service.(right click, it will freeze EM for few minutes and restart oms)
- Deploy plugin on all EM Agents 12c (right click)
For some security restriction reasons, you may want to download plugin from https://edelivery.oracle.com/ and manualy import plug-in to OMS.
- Update java tzdata. Theres is an issue in EM 12c.I have no idea why 12.1.0.2 packed with ancient JDK1.6.0_24, where there is an old time offset for Moscow +3. Current offset is +4. So, if You want to have correct time in EM and no time related issues - you should update java tzdata in all 12c agents and OMS. Download timezone updater tool with latest tzdata file included here: http://www.oracle.com/technetwork/java/javase/tzupdater-readme-136440.html
 As workaround you may run java processes with some key like -Duser.timezone=GMT+4 (correct for you country), or\and correct $AGENT_HOME/sysman/config/emd.properties for agents, and choose wrong country - in EM. But i think it's not best practice.
- Install JDK on every planned to observe host target (http://www.oracle.com/technetwork/java/javase/downloads/jdk6u38-downloads-1877406.html)

- Configure Golden Gate targets
  - add ENABLEMONITORING to GLOBALS parameter file an save it:

[oracle@test ~]$ cd $GG_HOME
[oracle@dbtest gg]$ ggsci
GGSCI (test.local) 1> edit params ./GLOBALS
CHECKPOINTTABLE GGS_CHECKPOINT
GGSCHEMA gguser
ENABLEMONITORING


- configure password for java agent

[oracle@dbtest gg]$ ./pw_agent_util.sh -jagentonly
Please create a password for Java Agent:

- configure jagent parameters in $GG_HOME/cfg/Config.properties

agent.type.enabled=OEM
jagent.rmi.port=5559
jagent.host=test.local
jagent.jmx.port=5555
jagent.username=user_name


-  create jagent datastore( Berkley DB) to store monitoring data and restart manager process:

[oracle@test ~]$ cd $GG_HOME
[oracle@dbtest gg]$ ggsci
GGSCI (dbtest.local) 1> create datastore
GGSCI (dbtest.local) 2> stop mgr
GGSCI (dbtest.local) 3> start mgr
GGSCI (dbtest.local) 4> start jagent


GGSCI (dbtest.local) 1> info all
Program     Status      Group       Lag at Chkpt  Time Since Chkpt
MANAGER     RUNNING
JAGENT      RUNNING
REPLICAT    RUNNING     RQIWIE      00:00:00      00:00:01
REPLICAT    RUNNING     RQIWIM      00:00:00      00:00:00
REPLICAT    RUNNING     RQIWITX     00:00:00      00:00:02

- Add Golden Gate instance to monitoring:

 -Setup->Add Target->Configure Auto Discovery->Edit Parameters.  Add JAgent Username,password and Jagent RMI Port=5559,which was previously specified in config.properties file on targets.
- Add target Setup->Add Target->Auto Discovery Results-> Choose Golden Gate and promote it.
- Done. You can see pictures like this:


Oracle Install Guide
Oracle® Enterprise Manager Oracle GoldenGate System Monitoring Plug-In Installation GuideRelease 12.1.0.1.1




Thursday, August 9, 2012

Reading alert.log in PL\SQL

In this short article I'll show simple block, to read errors in alert.log without having access to server itself. Be carefully, if log file big enough - script going to work forever. In my case - 300mb file parsed in approx 40 sec + 10 sec to output result. However, you may abort it anytime.
Let's find alert.log and create work directory. Also you neeed privilege to work with UTL_FILE:


SQL> select physical_filename,physical_path from v$diag_dir_ext where physical_filename like 'alert%.log'

SQL> create or replace directory TRACE as 'path_where_alert_log_is';



SQL> DECLARE
v_dir varchar2(50) := 'TRACE';
v_name varchar2(30) := 'alert_SID.log';
v_inputfile UTL_FILE.FILE_TYPE;
v_newline varchar2(500);
v_newline_1 varchar2(500);
v_offset number;
BEGIN
dbms_output.enable(10000000);
v_inputfile := UTL_FILE.FOPEN(v_dir,v_name,'r');
LOOP
BEGIN
utl_file.get_line(v_inputfile, v_newline);
IF substr(v_newline,1,4) = 'ORA-' 
 and substr(v_newline,1,9) not in ('ORA-00942','ORA-06550','ORA-12012') /* you may specify some expected errors here */
  then
dbms_output.put_line(v_newline);
utl_file.get_line(v_inputfile,v_newline_1);
dbms_output.put_line(v_newline_1);
end if;
END;
END LOOP;
EXCEPTION
WHEN NO_DATA_FOUND THEN
dbms_output.put_line('EXIT!');
UTL_FILE.FCLOSE(v_inputfile);
END;
/

The output would be something like: 
Sun Aug 05 06:00:43 2012
ORA-16957: SQL Analyze time limit interrupt
Sun Aug 05 06:04:44 2012
ORA-20011: Approximate NDV failed: ORA-29913: error in executing ODCIEXTTABLEOPEN callout
KUP-11024: This external table can only be accessed from within a Data Pump job.