AIX Most secure OS? Think not.

IBM’s Xforce published their new 2008 annual report. In it they had this chart:
xforce2008

Surprising is that IBM put’s one of their own OS’s near the bottom of the list. Some of my opinions are :

1. No one uses AIX that much, so no one looks for holes in the code.
2. Any one who uses AIX, doesn’t have it directly connected to the Internet.
3. It is so cost prohibitive to use, that people are looking at Solaris/Linux or Windows to run their business on.

But the funniest thing about this is the last I used AIX the following were still done on install by IBM:
1. telnet enabled
2. root logins allowed remotely
3. no ssh comes with the OS, you have to install a crappy “linux toolkit”, and then install another 10 different packages to get SSH enabled.
4. No RBAC
5. Syslog configuration does not exist
6. Root does not even have a password on install

Seems to me that IBM needs to fix some fundamental issues with their OWN OS before they can say it is not one of the “Most Vulnerable Operating Systems”.

The funniest issue with this is for MacOSX to be listed at the top, all most all of those require some one to actually run something on the machine with administrative privileges.

AIX LDAP Replication

a while ago I wrote about getting AIX authenticating to a SUN LDAP because of problems with AIX’s LDAP replication that I was having.. I forgot to mention that I “fixed” the AIX LDAP Replication problem. The solution was to put the FQDN of each “master” server in the /etc/hosts file with the IP address associated with it. It seems that IBM’s LDAP server is completely brain dead when it comes to DNS and LDAP, and would not replicate correctly unless I put the master server’s IP’s in the /etc/hosts.. So if you are having problems with getting replication to work with IBM’s LDAP server, try putting the entries in /etc/hosts and I bet it will work..

AIX LDAP to Sun LDAP in 10 Semi Easy Steps

I have been having some problems getting IBM’s Tivoli Directory Server to replicate properly. So I decided to try and see what it would take to move the IBM AIX machines from using IBM’s LDAP Server to using Sun’s LDAP Server. This is what I hope to accomplish:

  1. Setup Sun’s JES Directory Server on a Solaris 10 machine
  2. Configure the DS to have the AIX schema and objectclasses needed for user and group info
  3. Configure a AIX test machine to authenticate against the Sun LDAP Server
  4. Eventually move all AIX machines from the IBM DS to Sun DS, thereby having one set of servers that control all users/passwords for all Sun/Linux/AIX machines

Step 1: AIX Schema
The first thing I had to overcome is how AIX has 4 different ways of using LDAP for authentication. AIX 4.3.3 and AIX 5.1 used a non RFC2307 compliant schema. AIX 5.2 and AIX 5.3 can use this old schema or RFC2307, or a new one called RFC2307AIX, which combines the old with the new (there is also another one that I will not cover). When I originally setup the IBM LDAP I used the RFC2307AIX because it allows me to store ulimits and other info about AIX accounts in LDAP and not on each individual machine. But this also makes it harder to port these things over to Sun’s LDAP. Which lead me to yesterday’s afternoon adventure, creating a schema file that would work. I will link to my final copy below.

Step 2: AIX ObjectClasses
Second up was to create the objectclasses required for AIX Authentication, which consisted of creating the eAccount, AIXAccount,AIXaccessGroup, ibm-SecurityIdentities, container, and account. Some of these may not be needed for a fresh install, but I am trying to move entries from IBM’s LDAP to Sun’s LDAP with the least amount of editing an extremely huge ldap ldif export file. I will link to my final copy of this file below as well.

Step 3: Install JES DS
The third step was to download and install the DS 5.2.P4 on my fresh install of Solaris 10 U3 running on a Sparc machine. I did the custom install as I wanted to change the location of where it was installed to. The other thing I did is not load any sample data. Once the install was done, I ran the /usr/lib/ldap/idsconfig script to setup the DS. This is sort of how it went (copied from another doc):


It is strongly recommended that you BACKUP the directory server
before running idsconfig.

Hit Ctrl-C at any time before the final confirmation to exit.

Do you wish to continue with server setup (y/n/h)? [n] y
Enter the iPlanet Directory Server's (iDS) hostname to setup: ldap2
Enter the port number for iDS (h=help): [389] 389
Enter the directory manager DN: [cn=Directory Manager] cn=Directory Manager
Enter passwd for cn=Directory Manager :
Enter the domainname to be served (h=help): [ldap2.example.com] ldap2.example.com
Enter LDAP Base DN (h=help): [dc=example,dc=com] dc=example,dc=com
Enter the profile name (h=help): [default] default
Default server list (h=help): [192.168.1.2] 192.168.1.2
Preferred server list (h=help):
Choose desired search scope (one, sub, h=help): [one] one
The following are the supported credential levels:
1 anonymous
2 proxy
3 proxy anonymous
Choose Credential level [h=help]: [1] 2
The following are the supported Authentication Methods:
1 none
2 simple
3 sasl/DIGEST-MD5
4 tls:simple
5 tls:sasl/DIGEST-MD5
Choose Authentication Method (h=help): [1] 2

Current authenticationMethod: simple

Do you want to add another Authentication Method? n
Do you want the clients to follow referrals (y/n/h)? [n] y
Do you want to modify the server timelimit value (y/n/h)? [n] n
Do you want to modify the server sizelimit value (y/n/h)? [n] n
Do you want to store passwords in "crypt" format (y/n/h)? [n] y
Do you want to setup a Service Authentication Methods (y/n/h)? [n] n
Client search time limit in seconds (h=help): [30]
Profile Time To Live in seconds (h=help): [43200]
Bind time limit in seconds (h=help): [10]
Do you wish to setup Service Search Descriptors (y/n/h)? [n] n

Summary of Configuration

1 Domain to serve : example.com
2 Base DN to setup : dc=example,dc=com
3 Profile name to create : default
4 Default Server List : 192.168.1.2
5 Preferred Server List :
6 Default Search Scope : one
7 Credential Level : proxy
8 Authentication Method : simple
9 Enable Follow Referrals : TRUE
10 iDS Time Limit :
11 iDS Size Limit :
12 Enable crypt password storage : TRUE
13 Service Auth Method pam_ldap :
14 Service Auth Method keyserv :
15 Service Auth Method passwd-cmd:
16 Search Time Limit : 30
17 Profile Time to Live : 43200
18 Bind Limit : 10
19 Service Search Descriptors Menu

Enter config value to change: (1-19 0=commit changes) [0] 0

Enter DN for proxy agent:
[cn=proxyagent,ou=profile,dc=example,dc=com]
Enter passwd for proxyagent:
Re-enter passwd:

WARNING: About to start committing changes. (y=continue, n=EXIT) y

1. Changed passwordstoragescheme to "crypt" in cn=config.
2. Schema attributes have been updated.
3. Schema objectclass definitions have been added.
4. NisDomainObject added to dc=example,dc=com.
5. Top level "ou" containers complete.
6. automount maps: auto_home auto_direct auto_master auto_shared
processed.
7. ACI for dc=example,dc=com modified to disable self modify.
8. Add of VLV Access Control Information (ACI).
9. Proxy Agent cn=proxyagent,ou=profile,dc=example,dc=com added.
10. Give cn=proxyagent,ou=profile,dc=example,dc=com read permission for password.
11. Generated client profile and loaded on server.
12. Processing eq,pres indexes:
ipHostNumber (eq,pres) Finished indexing.
uidNumber (eq,pres) Finished indexing.
ipNetworkNumber (eq,pres) Finished indexing.
gidnumber (eq,pres) Finished indexing.
oncrpcnumber (eq,pres) Finished indexing.
automountKey (eq,pres) Finished indexing.
13. Processing eq,pres,sub indexes:
membernisnetgroup (eq,pres,sub) Finished indexing.
nisnetgrouptriple (eq,pres,sub) Finished indexing.
14. Processing VLV indexes:
example.com.getgrent vlv_index Entry created
example.com.gethostent vlv_index Entry created
example.com.getnetent vlv_index Entry created
example.com.getpwent vlv_index Entry created
example.com.getrpcent vlv_index Entry created
example.com.getspent vlv_index Entry created

idsconfig: Setup of iDS server ldap2 is complete.

Note: idsconfig has created entries for VLV indexes. Use the
directoryserver(1m) script on ldap2 to stop
the server and then enter the following vlvindex
sub-commands to create the actual VLV indexes:

directoryserver -s <server -instance> vlvindex -n userRoot -T example.com.getgrent
...much deleted...
directoryserver -s <server -instance> vlvindex -n userRoot -T example.com.getspent

Unfortunately the “directoryserver” command does not exist in Solaris 10, so i did the following:

cd /ldapserver/slapd-ldap2
./stop-slapd
./vlvindex -n userRoot -T example.com.getgrent
./vlvindex -n userRoot -T example.com.gethostent
./vlvindex -n userRoot -T example.com.getnetent
./vlvindex -n userRoot -T example.com.getpwent
./vlvindex -n userRoot -T example.com.getrpcent
./vlvindex -n userRoot -T example.com.getspent
./vlvindex -n userRoot -T example.com.getauhoent
./vlvindex -n userRoot -T example.com.getsoluent
./vlvindex -n userRoot -T example.com.getauduent
./vlvindex -n userRoot -T example.com.getauthent
./vlvindex -n userRoot -T example.com.getexecent
./vlvindex -n userRoot -T example.com.getprofent
./vlvindex -n userRoot -T example.com.getmailent
./vlvindex -n userRoot -T example.com.getbootent
./vlvindex -n userRoot -T example.com.getethent
./vlvindex -n userRoot -T example.com.getngrpent
./vlvindex -n userRoot -T example.com.getipnent
./vlvindex -n userRoot -T example.com.getmaskent
./vlvindex -n userRoot -T example.com.getprent
./vlvindex -n userRoot -T example.com.getip4ent
./vlvindex -n userRoot -T example.com.getip6ent
./start-slapd

(I installed the ldap server to /ldapserver)

Now that the indexes are created we can go on to the next step of modifying the schema.

Step 4: Importing new schema
Now I can import the AIXAttributes.ldif and the AIXObjectClasses.ldif files to my fresh newly installed server:


ldapmodify -D"cn=Directory Manager" -w MySuperSecretPass -h localhost -f AIXAttributes.ldif
ldapmodify -D"cn=Directory Manager" -w MySuperSecretPass -h localhost -f AIXObjectClasses.ldif

All should go well on the above imports. There were a couple of quirks that I found when creating the files, like how IBM uses one OID for a value when Sun uses a different one.

Step 5: Creating an OU for AIX data
By this time I have fired up the Console for the LDAP server and was doing things through the gui. The first thing I did was create a new OU for my aix data, i.e. ou=aixdata,dc=example,dc=com. This OU is where all my AIX stuff will be, the ou=people,dc=example,dc=com will be where my Sun users will go. (Can’t combine them yet because of massive amounts of differing UIDS between the AIX users and Sun Users).

After creating this, create 2 more OU’s under the aixdata. The first one will be for all users, ou=aixuser,ou=aixdata,dc=example,dc=com. The second will be for group information, ou=aixgroup,ou=aixdata,dc=example,dc=com.

Step 6: Create and AIX LDAP Admin Account
Because of how the AIX servers need to connect and get/put info into ldap, you will need to create an account that has read/write access to the 2 new ou’s you created. (This account is sort of similar to the Sun proxyagent account created with the idsconfig). I created my user, uid=aixldap,ou=aixdata,dc=example,dc=com. Also set the password for it to never expire (if you make it expire, you will have to update every AIX server every time the password expires). Once this user is created give it full read/write access to the ou=aixuser and ou=aixgroup with ACI’s.

Step 7: Export info from IBM LDAP
Since I am planning on moving from IBM LDAP to Sun LDAP I need to export the data from my IBM LDAP to an ldif format:


db2ldif -o /tmp/ldapexport.ldif

Do the above on the IBM LDAP server, it will put a file called /tmp/ldapexport.ldif. But now I need to “clean” some stuff out of it. Some of the stuff I need to remove is anything that is not User and Group related. For example I had IBM LDAP Replication setup so there are a ton of entries for that. (so the only thing you should have in your ldif file are entries for the dn’s like username=*,ou=aixuser,ou=aixdata,dc=example,dc=com and groupname=*,ou=aixgroup,dc=example,dc=com
)

Step 8: Import users and groups
Next I imported the cleaned file:


ldapadd -D"cn=Directory Manager" -w SuperSecret -f /tmp/ldapexport.ldif

Now all your users and groups should be in the new ldap server.

Step 9: Misc config in Sun LDAP
Some other stuff I did was add a couple indexes to the non-standard attributes that AIX uses:
groupname
username
hostsallowedlogin

The hostsallowedlogin allows us to put an entry in the persons LDAP entry to say which hosts they can log in to. If the attribute does not exist they can log in to any host that is served by this LDAP server. But if they have a value in this attribute, they can only log in to those host(s). (there is also a hostsdeniedlogin, which is the opposite of the hostsallowedlogin, if you want them to log in to every machine but one you can just populate that single host in to the hostsdeniedlogin).

Step 10: Configure AIX to talk to ldap server
One of the fall backs I don’t like about LDAP on AIX is that you have to have a local ldap user, and the ldap client software does not come with the base os. So you will have to install the ldap.client.adt and ldap.client.rte (probably don’t need the adt, but I install it anyways). During this install it usually creates the ldap user and group, which is never where we want it so either create an ldap group and ldap user before you install it, or after it is installed do the following, change the userid and groupid to what you want it to be in /etc/passwd and /etc/group and then run:


find / -user OLDID -exec chown -h ldap {} \;
find / -group OLDID -exec chgrp -h ldap {} \;

Now we can run the mksecldap command:

mksecldap -c -h 'ldap2.example.com' \
-a 'uid=aixldap,ou=aixdata,dc=example,dc=com' -p 'myp@ss' \
-d 'ou=aixdata,dc=example,dc=com' -n '389' -t '0' -T '100'

The above is all on one line. You must make sure the -t is set to 0, if it is not then you will get some weirdness that I will talk about later.

Once this is done, then you need to tell AIX that it is to look in LDAP for it’s info, you will need to edit another file /etc/security/user:

in the “default:” stanza, you will need to change the SYSTEM variable to LDAP, and add a registry variable with the value of LDAP.

I.E.:

default:
admin = false
login = true
su = true
daemon = true
rlogin = true
sugroups = ALL
admgroups =
ttys = ALL
auth1 = SYSTEM
auth2 = NONE
tpath = nosak
umask = 022
expires = 0
SYSTEM = LDAP
registry = LDAP

logintimes =
pwdwarntime = 0
account_locked = false
loginretries = 0
histexpire = 0
histsize = 0
minage = 0
maxage = 0
maxexpired = -1
minalpha = 0
minother = 0
minlen = 0
mindiff = 0
maxrepeats = 8
dictionlist =
pwdchecks =

Now you should be able to do an id username, and see some results:

AIXHost> id unixwiz
uid=106(unixwiz) gid=1(staff) groups=0(system),7(security),9(printq)

Next up try logging in remotely (you may have to create your home directory first). It should work.

Some Notes

  1. If you do not set the cachetimeout to be 0, if you change a users password as root, the user will get thrown in to a loop of changing their passwords. I.E. They will never be able to login as everytime they login it will say their password has expired and make them change it again and then it will kick them off (ie close their ssh session). Lather/rinse/repeat.
  2. After importing the users from an existing IBM server, you can delete the IBM-ENTRYUUID and the control attributes, they are for IBM DS only and have no use in Sun’s LDAP.
  3. Any problems on the import/etc, make sure to look at the ldap server error logs (/ldapserver/slapd-ldap2/logs/error_log) it should tell you exactly what is wrong.
  4. If you replicate this LDAP server, you should probably do a Master/Master relationship because of how AIX always stores info about last logins/etc in LDAP. I have not tested yet if it can follow referrals yet.

Here are the two files I was talking about:

AIXAttributes.ldif
AIXObjectClasses.ldif

Some other links:
IBM’s Redbook : Integrating AIX into Heterogeneous LDAP Environments, which I found was missing some stuff

Sun’s Cookbook for Solaris 8 client with Directory Server 5.1/Solaris 9 which I follow some times if I need to connect Solaris 8 machines in.

Hope this helps some one, if it does leave me a message.

IBM,Sun,LDAP

Interesting AIX tip

Ever wonder were all the space went in a file system? Does du and df show different results on the same file system? Chances are some one deleted a file that was open in a particular file system. For example the /tmp file system on one of our machines has been filling up and then some one was deleting files from it. But the space is never recovered. In the AIX errpt you may see something like this:


LABEL: JFS_FS_FULL
IDENTIFIER: 369D049B

Date/Time: Wed Sep 27 16:58:08 2006
Sequence Number: 780
Machine Id: 002AA9AF4C00
Node Id: aixbox
Class: O
Type: INFO
Resource Name: SYSPFS

Description
UNABLE TO ALLOCATE SPACE IN FILE SYSTEM

Probable Causes
FILE SYSTEM FULL

Recommended Actions
USE FUSER UTILITY TO LOCATE UNLINKED FILES STILL REFERENCED
INCREASE THE SIZE OF THE ASSOCIATED FILE SYSTEM
REMOVE UNNECESSARY DATA FROM FILE SYSTEM

Detail Data
MAJOR/MINOR DEVICE NUMBER
000A 0007
FILE SYSTEM DEVICE AND MOUNT POINT
/dev/hd3, /tmp

The hint to finding the files is given in the first line of the recommended actions. Use the fuser command. The actual command is: fuser -dV /tmp

What this will show you is something similar to this:

root@aixbox:/>fuser -dV /tmp
/tmp:
inode=34 size=675155 fd=0 200858
inode=43 size=114531 fd=0 286764
inode=66 size=59021846 fd=0 335986
inode=77 size=2322588 fd=0 389232
inode=46 size=601938 fd=0 413872
inode=61 size=28498 fd=0 430332
inode=44 size=1280774965 fd=0 434292
inode=40 size=2884063 fd=0 442590
inode=51 size=2395908 fd=0 467132
inode=73 size=8224333 fd=0 479402
inode=42 size=140607 fd=0 524474
inode=64 size=163405 fd=0 553054
inode=49 size=350562 fd=0 618644
inode=63 size=2375730 fd=0 663568
inode=74 size=3372392 fd=0 696356
inode=58 size=65535 fd=0 819204
inode=57 size=424777 fd=0 1106024
inode=62 size=2030397 fd=0 1147064
inode=76 size=57187 fd=0 1163494
inode=31 size=1376255 fd=0 1171540
inode=56 size=53834 fd=0 1216530
inode=52 size=361520961 fd=0 1278152
inode=81 size=15972886 fd=0 1294462
inode=70 size=13390097 fd=0 1323224
inode=60 size=7559 fd=0 1380400
inode=50 size=40132 fd=0 1429514
inode=65 size=720895 fd=0 1450220
inode=79 size=12582477 fd=0 1507350
inode=47 size=169682 fd=0 1593582
inode=48 size=259432 fd=0 1605642
inode=78 size=1488191 fd=0 1671280

The column we are really interested in is the last one. That is the column that contains the Process ID of the process that has a particular file open in that file system. So for example process id 1671280 has inode 78 in the /tmp file system open, and that file is using ~1.4MB of space. So if you “kill 1671280” you would recover the 1.4MB of space. (But make sure you kill it in the proper way, i.e. find out what the process really is and shut it down nicely.)

Interesting stuff in the mail today

Got a couple of magazines in the mail today. One is SysAdmin. Every once in a while they bundle something with it, todays bundle was a CDROM from IBM on “Migrating to Linux or AIX from Solaris OS”. It was one of their “RedBook” cdrom’s. Funny part to me is why would any one want to migrate from Solaris to Linux or AIX. I have been using AIX since about 1995, and to be honest, I would much rather use Solaris any day over AIX. The _only_ thing on AIX I “like” better is maybe printing. As for Linux, I have used it in the past, but in my own opinion it has fallen to it’s own success. Yeah a lot of people have learned how *NIX “works” by using it, but every day there is a distro coming out, and a new way of doing something. Basicly every distro has it’s own way of doing things and most of the time they don’t work on other distros. So for me I definately will not be migrating from Solaris to AIX or Linux, if any it would be the opposite.

The other magazine I got is the “IBM Systems Magainze”. It is funny reading through this “magazine” where everything is Pro-IBM and how much better it is than any thing else. One of the articles was “new” features to AIX 5.3, here are some of the things they list:

  • An environment variable LD_LIBRARY_PATH for runtime library path has been added for compatibility with SVR4 bases systems.

Interesting, that they say that as I have seen our DBA’s “misusing it” for years now. Doesn’t seem new to me..

Next up, in the base commands and libraries

  • A new flag has been added to the tar command, which would specifiy the list of files and/or directories to be excludede from the tar file being created,extracted or listed.
  • Text-processiong commands ed/vi/ex support unlimited line lengths
  • vi and ex can handle files up to 2GB
  • Support for unlimited number of fields and size of a line for awk
  • Cimand buffer sizes are unlimited, allowing an increased number of arguments o a command line

Seems AIX tar is a little behind in the game.. (Not to mention that I am still not sure if it can handle largerfiles. And the problem it has with it’s own userid nobody which is something to large to fit in uid_t in a tar archive.)

That is all cool about VI being able to handle 2Gb files, but how often do normal SA’s want to edit a 2gb file. (DBA’s, maybe all the time, but I do not try to ever edit a file that big)..

The unlimited line length is a definate plus, but wonder if there is a buffer overflow in that.

And the funniest of all, (you probably have to be an AIX administrator to get this, but it sounds funny if you are not).:

  • The PP size starts at 1MB and goes up to 128GB.

If you read it aloud around people who are not AIX adminstrators, they will probably start laughing. (For those who are not AIX administrators a PP is a physical partition size. Each logical volume in a volume group is made up of physical partitions. )

Here is a quick AIX LVM overview:
AIX LVM Overview

Basicly, we have a volume group called “testvg” that has 4 harddrives in it, (hdisk1, 2,3,4). In this volume group there are 3 Logical volumes created that are mounted to /filesystem1, /fs2 and /fs3. In this example /filesystem1 is made up of 4 PP’s, /fs2 is made up of 3 PP’s, and /fs3 is made up of 2 PP’s. They do not have to be in sequential order and if in fact when you change the size of file systems (usually increasing) it will grab the next free PP assuming there is one in the file system. The PP size is the “minimum” size a file system can be made and increased. So if I were to say that the PP size of each of the PP’s in the testvg were 512MB, then /filesystem1 would be roughly 2 GB, /fs2 would be 1.5 gb, and /fs3 would be 1 gig. This volume group is also not mirrored so the LV’s are just allocated in sequence. Maybe later I will do a more indepth picture out different options for AIX Volume Groups.

The “enhancements” to aix 53 for LVM include this:

AIX 5L Version 5.2 offers a new volume group type called scalable volume group (VG), which can accommodate up to 1,204 physical volumes,(N.B. old regular VG’s are limited to 32, Big VG’s it was increased but not sure to what) 4,096 logical volumes (LV’s) and 2,048K physical partitions (PPs). )

Needless to say you can make some pretty big volume groups with some pretty big file systems, but in my personal experience, the larger they are the longer they take to fix etc when a file system becomes corrupted, which for some reason seems to happen more with JFS2 than with JFS. Have not figured out what the exact cause is though.