AWS Mount Data Disk (Linux)

Amazon Web Services allows you to create EBS (Elastic Block Storage) instances which can be mounted to your EC2 Server instances. This allows an administrator to be able to size the storage solution effectively. These instructions should work on most Linux Flavours.

Use the lsblk command to view your available disk devices and their mount points (if applicable) to help you determine the correct device name to use.

[root ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvdf 202:80 0 100G 0 disk
xvda1 202:1 0 8G 0 disk /

The output of lsblk removes the /dev prefix from full device paths. In this example, /dev/xvda1 is mounted as the root device (note the MOUNTPOINT is listed as /, the root of the Linux file system hierarchy), and /dev/xvdf is attached, but it has not been mounted yet.

Determine whether you need to create a file system on the volume. New volumes are raw block devices, and you need to create a file system on them before you can mount and use them. Volumes that have been restored from snapshots likely have a file system on them already; if you create a new file system on top of an existing file system, the operation overwrites your data. Use the file -s device command to list special information, such as file system type.

[root ~]$ file -s /dev/xvdf
/dev/xvdf: data

If the output of the previous command shows simply “data” for the device, then there is no file system on the device and you need to create one. If you run this command on a device that contains a file system, then your output will be different.

[root ~]$ file -s /dev/xvda1
/dev/xvda1: Linux rev 1.0 ext4 filesystem data, UUID=1xxxxxx-exxx-4xxx-axxx-8xxxxxxxxxxxx (needs journal recovery) (extents) (large files) (huge files)

Use the following command to create an ext4 file system on the volume. Substitute the device name (such as /dev/xvdf) for device_name.

NOTE: This step assumes that you’re mounting an empty volume. If you’re mounting a volume that already has data on it (for example, a volume that was restored from a snapshot), don’t use mkfs before mounting the volume. Otherwise, you’ll format the volume and delete the existing data.

[root ~]$ mkfs -t ext4 device_name

To create a mount point directory for the volume. The mount point is where the volume is located in the file system tree and where you read and write files to after you mount the volume. Substitute a location for mount_point, such as /data.

[root ~]$ mkdir mount_point

To mount the volume at the location you just created.

[root ~]$ mount device_name mount_point

(Optional) To mount this EBS volume on every system reboot, add an entry for the device to the /etc/fstab file.

Format: device_name mount_point file_system_type fs_mntops fs_freq fs_passno

The last three fields on this line are the file system mount options, the dump frequency of the file system, and the order of file system checks done at boot time. If you don’t know what these values should be, then use the values in the following example for them (defaults,nofail 0 2). For more information on /etc/fstab entries, see the fstab manual page (by entering man fstab on the command line). use the UUID from the file -s device_name for the device_name. So for this example the entry would be:

[root ~]$ file -s /dev/xvdf
/dev/xvdf: Linux rev 1.0 ext4 filesystem data, UUID=2xxxxxx-5xxx-3xxx-1xxx-exxxxxxxxxxxx (needs journal recovery) (extents) (large files) (huge files)

[root ~]$ vi /etc/fstab

UUID=2xxxxxx-5xxx-3xxx-1xxx-exxxxxxxxxxxx /data ext4 default,nofail 0 2

Apache: DoS & DDoS Attack

Firstly, what is a DoS or DDoS Attack?  A DoS attack is a “Denial of Service” attack which typically is a computer being used to flood a target system or resource in an attempt to overwhelm it thus making the target system unable to service requests. A DDoS attack is a “Distributed Denial of Service” attack which are often global in nature and are distributed via botnets using multiple systems and/or resources to flood a target system. Both are bad.

Unfortunately, it was our turn to deal with a DoS attack.  Yesterday the phone rang with a client saying that there LAMP server was performing terribly. After a quick check the Apache httpd service was running hundreds of connections that were sucking the life out of the server.

A quick check of the connections yielded an IP address that did not seem to fit with the traffic pattern for the environment.

netstat -anp |grep ‘tcp\|udp’ | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n

It turned out that an IP address from the Netherlands was hammering the system.  So, as a quick fix, the IP address was added to the iptables deny list. The IP Tables Service restarted and the Apache Service restarted and the environment returned to normal operations.

Additional efforts should be taken to blacklist the IP addresses causing issues.

PeopleSoft – Some Users unable to open some pages

PeopleSoft – PSAPPSRV crash when some users try opening some pages

PeopleSoft – Personalization Corruptions

Not really too sure what title to use on this one.  One of my users started to report a problem that only she could replicate.  Every other user was able to login and navigate and open the page in question. We ended up cloning the user and the clone user had no issue opening the page.  Trace actually did not show the problem, but with a little creativity I was able to find the following case on MOS:

E-PIA: When A User navigate to some application pages, the Application Server Crash with Unrecoverable Exception Error. However, a Cloned User will work correctly (Doc ID 2033430.1)

The gift of the story turned out that the user had changed their personalizations or the personalizations had become corrupt. So I executed the following SQL:

DELETE FROM PSUSEROBJTYPE WHERE OPRID = ‘{PROBLEMUSER}’ AND MENUNAME = ‘CREATE_PAYMENTS’ AND PNLGRPNAME = ‘PYCYCL_DATA_INQ’;

You can delete all the personalizations for the problem user, but if you know the menu and component name you can isolate the specific personalization that is causing the problem. In this cause it was the Accounts Payable Pay Cycle Management Details Inquiry that was causing the one user all the issues.

Kill Application Engine (AE)

It amazes me that in the world of these high end databases that simple data handling can go for a crap simply because data was bulk loaded into the system.  Usually during big changes to an environment: data conversions, upgrades,  archiving or other aggressive changes, data will become out of sync with the statistics that are stored on the tables.

What are Database Statistics? In a nut shell it is data about data or  simply metadata. Oracle statistics is metadata about your data. There are several kinds of statistics in Oracle mainly: Object statistics, System statistics and fixed table statistics.

So if you have bad statistics your Application Engine may not be able to execute code correctly.  It will simply appear to be hung.  I have seen processes run for days and it will never return a result because of bad statistics.  After updating the statistics those exact same processes that ran for days will run in seconds, sometimes faster.  Okay, so now the catch is you have to “kill” the process in order for the process to be re-ran and pick up the correct statistics. This isn’t the easiest process in the world to do, because if you try to just cancel the job through the process scheduler it may or may not cancel nicely. However, if you “kill” the SQL at the database level that is hung the application engine for go to ERROR, and you can restart the AE from the last committed point that it executed to.  This is much cleaner than any other method I have found, especially if you have a huge amount of stuff already done by the application engine prior to running into this issue.  Upgrades are notorious for this, which is why they have “update statististics” steps all throughout the upgrade process now.

So in Oracle SQL we need to find the SQL that is causing the issue:

select * from v$session where program like ‘%PSAESRV%’;  –See SQL running by Application Engine

select * from v$sql where sql_id = ‘SQL_ID from v$Session’;  — Confirm you have the write SQL

alter system kill session ‘{sid},{serial#}’;

It may take a few minutes to find the SQL causing the problems, but don’t worry, it will still be running after you find it.  Once it is killed, the AE will go to error, and you can restart it.  It will start up at the last commit point, which you can usually see if you query the AE Control table:

select * from ps_aeruncontrol;

Rerun your statistics for the schema or for the tables in question.

exec dbms_stats.gather_schema_stats(‘SYSADM’, cascade=>true);

Good luck out there!

Linux Curl with Proxy

What is Curl?  curl is a tool to transfer data from or to a server, using one of the supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET and TFTP). The command is designed to work without user interaction.

So in the cloud you may find yourself on a server that needs to communicate to an external system but it isn’t allowed to go there directly.  In which case you will most likely have a proxy server to communicate through.  In PeopleSoft you can define a web gateway proxy server, however, to test to make sure that the server is working correctly with the proxy you will want to do a curl test.

Linux Command Prompt –> curl http://whatsmyip.com

This should return the external IP address of the system you are coming out of.  If this returns nothing you are likely blocked from going to that address.  So to push the request through the proxy server, you need to set the proxy in an environment variable:

Linux Command Prompt –> export http_proxy=http://myproxy.atmyserver.com:80

Linux Command Prompt –> curl http://whatsmyip.com

Now as long as the proxy is allowed to communicate to that address you should see the IP address of the system you are communicating from.