Alvin's HISTFILE workaround

Here is the background of my story. In my site, there are 3 hosts (named host-A, host-B & host-C) running AIX 6.1. Each host allows a maximum of 2 telnet login sessions. So, I can at most login 6 telnet sessions.  My $HOME directory is mounted in host-B. Using NFS, all the 3 hosts will share the same $HOME directory. Ideally I would expect each telnet session have its own command history, but this does not happen.

Once login, the korn shell will create a command history file .sh_history in the $HOME.  Then the 6 telnet sessions will share the SAME command history file $HOME/.sh_history. This is really a nightmare.

To solve this problem, I tried to define the HISTFILE variable in the first line of my $HOME/.profile.  This HISTFILE will define an unique filename for each of the 6 telnet sessions. So, each session will have different command history filenames. However, such a HISTFILE setup does not work.

Why the HISTFILE is not functioning? Sometimes even the HISTFILE is defined in the first line of the $HOME/.profile file, it is not functioning at all. Why? It is because of the /etc/profile file. When korn shell is your login shell, once login, the korn shell will firstly execute the system file /etc/profile before executing the $HOME/.profile file. Usually, some third party software (such as file transfer or database software) will modify this /etc/profile to instruct it to run some setup scripts of the software. Commonly those setup script will use some functions. Once there is function declaration in the script, the korn shell will record the function in the command history. Thus, the default command history file $HOME/.sh_history is used. Once the default $HOME/.sh_history file is used, the name of the command history file will not be changed by the HISTFILE variable.

Actually, the command [[ set -o nolog ]] can prevent the korn shell to record the function the command history file. Usually, the nolog option will not be set in the /etc/profile. You can modify the /etc/profile to add this nolog option. However, once the host is already in production, changing the system file /etc/profile requires numerous procedures and approvals. No one will take the risk to approve this kind of system file modifications. In other words, it is very difficult to make this change in a production host. As a result, those third party software setup script and its function in the /etc/profile will force the korn shell to use the command history file $HOME/.sh_history once you login.

To be more complicate, the AIX 6.1 seems cache the command history in the memory. Therefore, when issuing the command [[ fc -l ]], sometimes the result is the command history in the cache memory but sometimes it is a mix of commands from the 6 telnet sessions due to sharing same $HOME/.sh_history file.

The korn shell in AIX 6.1 has a built-in fc command. When issuing the command [[ fc -l ]] in an interactive korn shell, this built-in fc command seems rely on the cached command history.

However, when using [[ fc -l ]] inside a script (which is run under another process), the external /usr/bin/fc or /bin/fc commands may be used (because a process should not use the korn shell process built-in commands). For the external command /usr/bin/fc, it seems that it cannot access the cached command history of the login korn shell. So, it will reply on the default command history file $HOME/.sh_history or the HISTFILE environment variable. When the HISTFILE is defined and exported to the script, it will use it, even though this HISTFILE is not functioning in the login korn shell. This further complicates the situation.

Also, when issuing [[ fc -l & ]] or [[ nohup fc -l & ]] in an interactive korn shell, a separate process is run for the command. This separate process seems behave quite similar to the situation above.

In order to solve the above problems, what we need is an unique command history file for each of the 6 telnet session. But, how to do so ? Here is my solution. I named it as Alvin's HISTFILE workaround.

This workaround is to add the following line at the beginning of the $HOME/.profile file.

if tty -s then      export HISTFILE=~/.sh_history.e`/bin/date +"%s"`.`/bin/hostname`.pts`/bin/tty | /bin/awk -F'/' '{print $NF}'`.r$RANDOM      if [ -e ~/.sh_history ]      then           /bin/ln ~/.sh_history $HISTFILE           /bin/rm -f ~/.sh_history           /bin/find ~ -type f -name '.sh_history.e*r*' -mtime +15 -exec /bin/rm {} \; &      fi fi

Here is a detail description of how this workaround achieve my purpose.

When I telnet login the first session, this workaround is executed

The first line [[ if tty -s ]] is to ensure that this is an interactive shell. In my site, there are scheduled jobs running using root-id to [[ su - ]] to my user-id to execute some programs on behave of me. This action will call my $HOME/.profile and then run some programs in "no terminal" mode. In AIX 6.1, such a "no terminal" situation will make the [[ pts ]] command to give error messages which will spoil the scheduled jobs and also the HISTFILE variable. Anyway, the primary purpose of the workaround is for interactive korn shell, so it is better to have such a [[ if tty -s ]] checking.

The next line is to define the HISTFILE variable. The variable will use the epoch time, the hostname, the pts number and a random number. So, for the 6 telnet login sessions, the HISTFILE will be unique.

Then, the [[ if ]] statement is to double confirm the $HOME/.sh_history is already exist. Such file should be created by the /etc/profile once login.

Then, we use hard-link to make the two files $HOME/.sh_history and the $HISTFILE sharing the same i-node. Now, the content of this i-node file is the command history of this interactive korn shell. This i-node has 2 links : the $HOME/.sh_history and the $HISTFILE files. Now, no matter whether the built-in fc command or the external /usr/bin/fc command is used, each command will refer to the same command history content in that i-node.

The next line is to remove the filename .sh_history in the $HOME directory. Such a file removal will not affect the interactive korn shell. The shell should already get the i-node number for command history processing throughout its life. The i-node will not physically clean-up because there is still a link (the $HISTFILE file) pointing to it. As a result, both the interactive korn shell and the external /usr/bin/fc commands will function as usual.

Now, I login the second telnet session. Since the file .sh_history is already removed from the $HOME directory, the /etc/profile will create a new .sh_history file (with a new i-node number, of course). Then, this second interactive korn shell will work on this new i-node number for command history. Also, a new HISTFILE is defined for this shell. The $HISTFILE filename is different from the first telnet session. So, this new i-node and new $HISTFILE is only for this second telnet shell. Therefore, I have a completely independent command history file in the second login shell.

This process is repeated for the third, fourth, fifth and sixth interactive login korn shell. There will be six $HISTFILE in the $HOME directory, each using a different i-node.

The final [[ find ]] command is to clean-up those $HISTFILE files older than 15 days. It is assumed that you exit the login korn shell at least 15 days after login. If you do not exit the shell after 15 days, a new login process will kill your $HISTFILE and your i-node will be clean-up. Then, un-expected behavior will be resulted. Normally, you should exit the shell as soon as you no longer need it.

Alvin SIU
2013-04-23

Copyright/Licence Information: All information and coding in this article is offered at no charge for NON-COMMERCIAL PERSONAL USE only. This blog and the coding is copyright. Reproduction of this blog and its coding in whole or in part in paper or digitally or in any other forms without the explicit written permission of the author is strictly prohibited.

Disclaimer: All information in this article is distributed "as is" and is UNSUPPORTED. NO WARRANTY of any kind is expressed or implied. You use AT YOUR OWN RISK. The author will not be liable for any data loss, damages, and loss of profits or any other kind of tangible or intangible loss while using or misusing wholly or partly of the information.

Use CamStudio as screen recorder

This web page [ How to use CamStudio ] teach you how to use the software to record the screen.

MS Data Access Components

Mahesh Chand has written an article [[ Micrsoft Data Access Components (MDAC) ]] to explain various Microsoft database access methods and which one to use and when.

This article [[ Microsoft Jet Database Engine ]] in Wikipedia says that the Access ODBC developers used parts of the Jet code to produce the ODBC driver.

---

Access97 mdb in WinXP

Recently, I was asked to migrate an application from the WinNT4 world to the WinXP world. Here is some details of the application.

	Old World	New World
OS of the PC	WinNT4 WS SP6	WinXP Pro SP3
The Application is written by	Delphi 4	N/A
Database Connectivity	ODBC	ODBC
Database	Access 97 mdb file	Access 97 mdb file

My first question is : Do I need to install Access97 in WinXP ? This article [[ Microsoft Access in Wikipedia ]] says that Access97 is not supported by the WinXP OS. Also, there is significant transition was from the Access 97 to the Access 2000 format.

After searching the net, it seems that there are many problems when running Access97 in WinXP. For example, this article [[ Access 97 Install Difficulty on Windows XP - System.mdw ]] shows a life experience about the difficulties for installing Access97 in WinXP.

Is it possible to setup ODBC driver for the Access97 mdb file in WinXP ?. This post [[ Access97 ODBC ]] is doing this thing.

This article [[ Microsoft Jet Database Engine ]] in Wikipedia says that :

The Jet DLL determined what sort of database it was accessing, and how to perform what was requested of it. If the data source was an MDB file (a Microsoft Access format) then it would directly read and write the data to the file. If the data source was external, then it would call on the correct ODBC driver to perform its request.

---

Unicode Related Articles

This blog article [ 中文化和国际化问题权威解析之一：字符编码发展历程 ] give a clear and precise description on Unicode with a brief history of character encoding. The article is written in Simplified Chinese.

---

ASCII, Code Page and Unicode

This web page [ Microsoft Typography | Developer information | Character sets ] has documented the history of character sets from ASCII, OEM 8-bit character, Windows ANSI and code page, DBCS to Unicode. At the end of the web page, it also tells 12 steps to Unicode-enabling.

---

Unicode and UTF8

What is the relationship between Unicode and UTF-8 ? This article "What Is UTF-8 And Why Is It Important?" is quite good for reading. Anyway, I want to further explain a little bit.

Actually, the so-called Unicode for most of the time is referring to the mapping table which map a number (called code point) to a character. The number can be a small number between 0 to 127. This number will map to the normal ASCII characters. For example, the decimal number 65 (hex 41) map to English letter A.

The number can be a little larger, which need 16-bit to represent it. Then, this number will map to, such as, some commonly used Chinese character. For example, the decimal number 20,013 (hex 4E2D) map to Chinese character 中.

If the number is further larger (which need 20-bit to represent it), this number will map to, such as, some rarely used Chinese characters. For example, the decimal number 194,712 (hex 2F898) map to Chinese character 𦇚

So far, I am just talking about the mapping and the size (16-bit & 20-bit) of the number. I have not mentioned how to store this number.

At the first glance, you can simply store each character as a 20-bit number. But this is a waste of storage because there are many unused ZEROs. For example, the English letter A is ASCII-65. Its 1-byte representation is 0x41. Using 32-bit representation will be 0x00000041. There are many unused ZEROs at the beginning. For Chinese characters, those commonly used characters only need 2 bytes. If using 32-bit representation, the first 16-bit will also become unused ZEROs

To tackle this problem, a method UTF-8 is developed. This method will store the Unicode code point number in 1-byte, 2-byte or 3-byte, depending on the VALUE of the number. If the code point is a small number, i.e. those ASCII characters code points, only 1 byte will be used. About 1900+ code points (usually for those European letters) need 2 bytes. For those code points for commonly used Chinese characters, 3-byte is used.

It can be said that, under the UTF-8 method, the Unicode character table is divided into several areas. Those characters in the area related to ASCII will use 1-byte to store. About 1900+ characters in another area will use 2-byte to store. Most commonly used Chinese characters will use 3-byte to store.

Take an experiment, cut and paste the following line to PSPad :

abcdefghijαβγδ中文字

Then, choose Format > UTF-8 and then save the file.

The file size is 27 bytes. The 10 English letters consume 10 x 1 = 10 bytes. The 4 Greek letters consume 4 x 2 = 8 bytes. The 3 Chinese characters consume 3 x 3 = 9 bytes. 10 + 8 + 9 = 27 bytes. You can open the file using Firefox 3.0.8 which can auto-detect UTF-8 to open the file.

(By the way, you can save it using UTF-16 LE encoding by chooseing Format > UTF-16 LE. The file size will be 36 byte. The 17 characters consumes 17 x 2 = 34 bytes. Plus the BOM marker FFFE total to 2 + 34 = 36 bytes. Firefox can also auto-detect UTF-16 LE to open the file.)

If simply looking at Chinese characters, this UTF-8 method seems quite "expensive" because it uses 3 bytes to store those commonly used Chinese characters. And, those Chinese characters is a 16-bit (2-byte) code point. In other words, there is a 8-bit (1-byte) overhead (i.e. 50% overhead) for each Chinese character. Yes, this is the truth. Therefore, if the information is a pure Chinese character, using UTF-8 will make the file size much larger than using UTF-16 !

But, UTF-8 has many advantages.

First of all, UTF-8 is backward compatible to all ASCII encoding. The ASCII can be treated as a subset of UTF-8. All English characters are represented using 1 byte, the same as ASCII. Therefore, all existing software or else handling ASCII can also handle UTF-8.

Secondly, there is no unused ZEROs in UTF-8. In C programming language, the null character ZERO is the string terminator. If there is such a ZERO in the stream, it will terminate the string. Then, this unused ZERO will make many C program behave not as expected. UTF-8 encoding method ensure that there is no unused ZEROs. Many C program can run as usual.

Also, UTF-8 can define the character BOUNDARY. In UTF-8, if the code point is for ASCII character, it is stored as 0zzzzzzz (where zzzzzzz is the ASCII value ranging from 0 to 127). For those code point needing 2-byte representation, it is stored as 110yyyzz 10zzzzzz (where yyyzzzzzzzz is the code point). For those code point (e.g. for Chinese character) which need a 3-byte representation, it is stored as 1110yyyy 10yyyyzz 10zzzzzz . In other words, all 16-bit Unicode character is either a (0-something) or (110-something 10-something) or (1110-something 10-something 10-something) representation in UTF-8. For a more technical details, please refer to the web page UTF-8 in Wikipedia.

Furthermore, in information transmission, UTF-8 can help detecting error. For example, if a new character is beginning with 1110-something, 3-byte is expected. There should be two 10-something bytes follows. If you cannot find 2 more 10-something bytes, there must be something wrong. For another example, a 2-byte representation is (110-something 10-something). It is impossible to have this sequence (110-something 0-something). There must be something wrong.

At last, although this article uses only 1-byte, 2-byte and 3-byte as example, in theory, UTF-8 can be expanded into 4-byte, 5-byte, ... etc. For example, the 4-byte representation will be 11110xxx 10xxyyyy 10yyyyzz 10zzzzzz. This 4-byte representation can denote 21-bit Unicode characters.

Other useful references:

1. UTF-8, UTF-16, UTF-32 & BOM
   

---