Labels

Wednesday 30 December 2015

how to get live stream from camera using ffmpeg and vlc?

Write live video to mp4 file:
ffmpeg -f video4linux2 -s 320x240 -i /dev/video0 test.mp4
With sound:
ffmpeg -f video4linux2 -s 320x240 -i /dev/video0 -f alsa -i hw:0 -f mp4 test3.mp4
Write to a raw .264 file
ffmpeg -f video4linux2 -s 320x240 -i /dev/video0 -vcodec libx264 -f h264 test.264
With access unit delimiters
ffmpeg -f video4linux2 -s cif -i /dev/video0 -x264opts slice-max-size=1400:vbv-maxrate=512:vbv-bufsize=200:fps=15:aud=1 -f mp4 -vcodec libx264 -f h264 rtp_aud_.264
Convert raw .264 to mp4
ffmpeg -i test.264 test_convert.mp4
Pipe into another process
ffmpeg -f video4linux2 -s cif -i /dev/video0 -x264opts slice-max-size=1400:vbv-maxrate=512:vbv-bufsize=200:fps=15 -f mp4 -vcodec libx264 -f h264 pipe:1 | ./pipe_test 
Send over RTP and generate SDP on std out:
ffmpeg -f video4linux2 -s cif -i /dev/video0 -x264opts slice-max-size=1400:vbv-maxrate=512:vbv-bufsize=200:fps=15 -f mp4 -vcodec libx264 -f rtp rtp://127.0.0.1:49170
Generates:
v=0
o=- 0 0 IN IP4 127.0.0.1
s=No Name
c=IN IP4 127.0.0.1
t=0 0
a=tool:libavformat 54.17.100
m=video 49170 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1
Low Delay RTP:
ffmpeg -f video4linux2 -s cif -i /dev/video0 -x264opts slice-max-size=1400:vbv-maxrate=100:vbv-bufsize=200:fps=25 -f mp4 -vcodec libx264 -tune zerolatency -f rtp rtp:127.0.0.1:49170
Can be played with (where test.sdp should include the generated SDP):
./vlc -vvv test.sdp
Using VLC to stream over RTP AND display live stream:
./vlc -vvv v4l2:///dev/video0 :v4l2-standard= :v4l2-dev=/dev/video0 --v4l2-width=352 --v4l2-height=288 --sout-x264-tune zerolatency --sout-x264-bframes 0 --sout-x264-aud --sout-x264-vbv-maxrate=1000 --sout-x264-vbv-bufsize=512 --sout-x264-slice-max-size=1460 --sout '#duplicate{dst="transcode{vcodec=h264,vb=384,scale=0.75}:rtp{dst=130.149.228.93,port=49170}",dst=display}'
However this does not seem to generate the SPSs and PPSs.

Edit:
 --sout-x264-options=repeat-headers is necessary to repeat SPS and PPS in stream.

Low delay capture with VLC:
./vlc --live-caching 0 --sout-rtp-caching 0 -vvv v4l2:///dev/video1 :v4l2-standard= :v4l2-dev=/dev/video0 --v4l2-width=352 --v4l2-height=288 --sout-x264-tune zerolatency --sout-x264-bframes 0 --sout-x264-options repeat-headers=1 --sout-x264-aud --sout-x264-vbv-maxrate=1000 --sout-x264-vbv-bufsize=512 --sout-x264-slice-max-size=1460 --sout '#duplicate{dst="transcode{vcodec=h264,vb=384,scale=0.75}:rtp{dst=130.149.228.93,port=49170}",dst=display}'

Tuesday 29 December 2015

mpeg-ts outline specification

Mpeg-ts outline specification:

1. Introduction
    1.1. Packetized elementary streams (PES)
2. Time Stamp
    1.3. Mpeg transport stream (MPEG-TS)
2. Multiplexing
3. De-Multiplexing
    3.1. Audio-Video synchronization
4. Results
    4.1. Buffer fullness
    4.2. Synchronization and skew calculation
5. Conclusions

1. Introduction


H.264[5,48,51] is the latest and the most advanced video codec available today. It was jointly developed by the VCEG (video coding experts group) of ITU-T (international telecommunication union) and the MPEG (moving pictures experts group) of ISO/IEC (international standards organization). This standard achieves much greater compression than its predecessors like MPEG-2 video[37], MPEG4 part2 visual[38] etc. But the higher codingefficiency comes at the cost of increased complexity. The H.264 has been adopted as the video standard for many applications around the world including ATSC[21]. H.264 covers only video coding and is not of much use unless the video is accompanied by audio. Hence it is relevant and practical to encode/decode and multiplex/demultiplex both video and audio for replay at the receiver.
HEAACv2[49,50] or High efficiency advanced audio codec version 2 also known as enhanced aacplus is a low bit rate audio codec defined in MPEG4 audio profile[2] belonging to the AAC family. It is specifically designed for low bit rate applications such as streaming, mobile broadcasting etc. HE AAC v2 has been proven to be the most efficient audio compression tool available today. It comes with a fully featured toolset which enables coding in mono, stereo and multichannel modes (up to 48 channels). HEAACv2[7] is the adopted standard for ATSC-M/H and many other systems around the world.
The encoded bit streams or elementary streams of H.264 and HEAACv2 are arranged as a sequence of access units. An access unit is a coded representation of a frame. Since each frame is coded differently the size of each access unit also varies. In order to transmit a multimedia content (audio and video) across a channel, the two streams have to be converted in to a single stream of fixed sized packets. For this the elementary streams have to undergo two layers of packetization (Fig. 1). The first layer of packetization yields Packetized Elementary Stream (PES) and the second layer of packetization where the actual multiplexing takes place results in a stream of fixed sized packets called as Transport Stream (TS). These TS packets are what are actually transmitted across the network using broadcast techniques such as those used in ATSC and DVB[16].
http://article.sapub.org/image/10.5923.j.ajsp.20120203.03_001.gif

1.1. Packetized elementary streams (PES)

PES packets are obtained after the first layer of packetization of coded audio and coded video data. This packetization process is carried out by sequentially separating out the audio and video elementary streams into access units. Hence each PES packet is an encapsulation of one frame of coded data. Each PES packet contains a packet header and the payload data from only one particular stream. PES header contains information which can distinguish between audio and video PES packets. Since the number of bits used to represent a frame in the bit stream varies (for both audio and video) the size of the PES packets also varies. Figure 2 shows how the elementary stream is converted into PES stream.
The PES header format used is shown in table 1. The PES header starts with a 3 byte packet start code prefix which is always “0x000001” followed by 1 byte stream id. Stream id is used to uniquely identify a particular stream. Stream id along with start code prefix is known as start code (4 bytes). PES packet length may vary and go up to 65536 bytes. In case of longer elementary stream, the packet length may be set as unbound i.e. 0, only in the case of video stream. The next two bytes in the header is the time stamp field, which contains the playback time information. In the proposed method, frame number is used to calculate the playback time, which is explained next. 

2. Time Stamp

Time stamps indicate where a particular access unit belongs in time. Audio-video synchronization is obtained by incorporating time stamps into the headers in both video and audio PES packets.
Traditionally to enable the decoder to maintain synchronization between audio track and video frames, a 33 bit encoder clock sample called Program Clock Reference (PCR) is transmitted in the adaptation field of the TS packet from time to time (every 100 ms). This along with the presentation time stamp (PTS) field that resides in the PES packet layer of the transport stream is used to synchronize the audio and video elementary streams.

The proposed method uses the frame numbers of both audio and video as time stamps to synchronize the streams. As explained earlier both H.264 and HE AAC v2 bit streams are organized into access units i.e. frames separated by their respective sync sequence. A particular video sequence will have a fixed frame rate during playback which is specified by frames per second (fps). So assuming that the decoder has a prior knowledge about the fps of the video sequence, the presentation time (PT) or the playback time of a particular video frame can be calculated using (1).

The AAC compression standard defines each audio frame to contain 1024 samples per channel. This is true for HE AAC v2[2,3,7] as well. The sampling frequency of the audio stream can be extracted from the sampling frequency index field of the ADTS header. The sampling frequency remains the same for a particular audio stream. Since both samples per frame and sampling frequency are fixed, the audio frame rate also remains constant throughout a particular audio stream. Hence the presentation time (PT) of a particular audio frame (assuming stereo) can be calculated as follows:
(2)

The same expression can be expanded for multi channel audio streams, just by multiplying the number of channels.

 Table 1. PES header format[4]

NameSize (in Bytes)Description
Packet start code prefix30x000001
Stream id1Unique ID to distinguish between audio and video PES packetExamples: Audio streams (0xC0-0xDF), Video streams (0xE0-0xEF)[3]
Note: the above 4 bytes together are known as start code.
PES Packet length2The PES packet can be of any length. A value of zero for the PES packet length can be used only when the PES packet payload is a video elementary stream
Time Stamp2frame number
Once the presentation time of one stream is calculated, the frame number of the second stream that has to be played at that particular time can calculated. This approach is used at the decoder to achieve the audio-video synchronization or lip synchronization; this is explained in detail later on.
Using frame numbers as time stamps has many advantages over the traditional PCR approach. Obvious advantages are that there is no need to send the additional Transport Stream (TS) packets with PCR information, reduced overall complexity, no need to consider clock jitters during synchronization, smaller time stamp field in the PES packet i.e., just 16 bits to encode frame number compared to 33 bits for the Presentation Time Stamp (PTS) which has a sample from the encoder clock. The time stamp field in this project is encoded in 2 bytes in the PES header, which implies that time stamp field can carry frame numbers up to 65536. Once the frame number of either stream exceeds this number, which is a possibility in the case of long video and audio sequences, the frame number is reset to 1. The reset is done simultaneously on both audio and video frame numbers as soon as the frame number of either one of the stream crosses 65536. This will not create a frame number conflict at the de-multiplexer during synchronization because the audio and video buffer sizes are much smaller than the maximum allowed frame number.

1.3. Mpeg transport stream (MPEG-TS)

PES packets are of variable sizes and are difficult to multiplex and transmit in an error prone network. Hence they undergo one more layer of packetization which results in Transport Stream (TS) packets.
MPEG Transport Streams (MPEG-TS)[4] use a fixed length packet size and a packet identifier identifies each transport packet within the transport stream. A packet identifier in an MPEG system identifies the type of packetized elementary stream (PES) whether audio or video. Each TS packet is 188 bytes long which includes header and payload data. Each PES packet may be broken down into a number of transport stream (TS) packets since a PES packet which represents an access unit (a frame) in the elementary stream which is usually much larger than 188bytes. Also a particular TS packet should contain data from only one particular PES. The TS packet header (Fig. 3) is three bytes long; it has been slightly modified from the standard TS header format for simplicity, although the framework remains the same.

The sync byte (0x47) indicates the start of the new TS packet. It is followed by a payload unit start indicator (PUSI) flag, which when set indicates that the data payload contains the start of new PES packet. The Adaptation Field Control (AFC) flag when set indicates that all the allotted 185 bytes for the data payload are not occupied by the PES data. This occurs when the PES data is less than 185 bytes. When this happens the unoccupied bytes of the data payload are filled with filler data ( all zeros or all ones), and the length of the filler data is stored in a byte called the offset right after the TS header. Offset is calculated as 185 – length of PES data. The Continuity Counter (CC) is a 4 bit field which is incremented by the multiplexer for each TS packet sent for a particular stream I.e. audio PES or video PES, this information is used at the de-multiplexer side to determine if any packets are lost, repeated or is out of sequence. Packet ID (PID) is a unique 10 bit identification to describe a particular stream to which the data payload belongs in the TS packet.

2. Multiplexing

Multiplexing is a process where Transport Stream (TS) packets are generated and transmitted in such a way that the data buffers at the decoder (de-multiplexer) do not overflow or underflow. Buffer overflow or underflow by the video and audio elementary streams can cause skips or freeze/mute errors in video and audio playback.
The flow chart of the proposed multiplexing scheme is shown in figures 4 and 5. The basic logic is based on both audio and video sequences having constant frame rates. For video, the number of frames per second value will remain the same throughout the video sequence. In an audio sequence since sampling frequency remains constant throughout the sequence and samples per frame is fixed (1024 for stereo), the frame duration also remains constant.
http://article.sapub.org/image/10.5923.j.ajsp.20120203.03_005.gif
 
For transmission a PES packet which represents a frame is logically broken down to n (n depends on PES packet size) number of TS packets of 188 bytes each. The exact presentation time of each TS packet (PTAudio/VideoTS) may be calculated as shown in (3) through (8), where NTSVideo/Audio is the number of TS packets required to represent corresponding PES packet or frame:
(3)
(4)
(5)

Similarly for audio:
(6)
(7)

Whereis given by
(8)

From (5) and (8) it may be observed that the presentation time of a current TS packet is the cumulative sum of presentation time of previous TS packet (of the same type) and the current TS duration. The decision to transmit a particular TS packet (audio or video) is made by comparing their respective presentation times. Whichever stream has a lower value; it is scheduled to transmit a TS packet. This makes sure that both audio and video content get equal priority and also transmitted uniformly. Once the decision about which TS to transmit is made, the control goes to one of the blocks where the actual generation and transmission of TS and PES packets take place.
 
http://article.sapub.org/image/10.5923.j.ajsp.20120203.03_014.gif
In the audio/video processing block (Fig. 5), the first step is to check whether the multiplexer is still in the middle of a frame or in the beginning of a new frame. If a new frame is being processed, (4) or (7) is executed appropriately, to find out the TS duration. This information is used to update the TS presentation time at a later stage. Next data is read from the concerned PES packet, if PES is larger than 185 bytes then only the first 185 bytes are read out and the PES packet is adjusted accordingly. If the current TS packet is the last packet for that PES packet, a new PES packet for the next frame (for that stream) is generated. Now the 185 bytes payload data and all the remaining information are ready to generate the transport stream (TS) packet.
Once a TS packet is generated, the TS presentation time is updated using (5) and (8). Then the control goes back to the presentation time decision block and the entire process is repeated till all the video and audio frames are transmitted.
 

 http://article.sapub.org/image/10.5923.j.ajsp.20120203.03_015.gif

3. De-Multiplexing

The Transport Stream (TS) input to a receiver is separated into a video elementary stream and audio elementary stream by a de-multiplexer. At this time, the video elementary stream and the audio elementary stream are temporarily stored in the video and audio buffers, respectively. 
The basic flow chart of the de-multiplexer is shown in the figure 6. After receiving a TS packet, it is checked for the sync byte (0X47), to check if the packet is valid or not. If invalid that packet is skipped and de-multiplexing is continued with the next packet. The valid TS packet header is read to extract fields like packet ID (PID), adaptation field control flag (AFC), payload unit start (PUS) flag, 4 bit continuity counter etc. Now the payload is prepared to be read into the appropriate buffer. By checking the AFC flag it can be known that an offset value has to be calculated or all 185 bytes in the TS packet have payload data. If the AFC is set then the payload is extracted by skipping the stuffing bytes.

The Payload Unit Start (PUS) bit is checked to see if the present TS packet contains a PES header. If so then, the PES header is first checked for the presence of the sync sequence (i.e. 0X000001). If not, the packet is discarded and the next TS packet is processed. If valid then the PES header is read and fields like stream ID, PES length, frame number are extracted. Now the PID is checked to see if it is an audio TS packet or video TS packet. Once this decision is made, the payload is written into its respective buffer. If the TS packet payload contained the PES header, information like frame number, its location in the corresponding buffer, PES length are stored in a separate array variable which is later used for synchronizing the audio and video streams.

Once the payload has been written into the audio/video buffer, video buffer is checked for fullness. Since video files are always much larger than audio files, the video buffer gets filled up first. Once the video buffer is full, the next occurring IDR frame is searched in the video buffer. Once found, the IDR frame number is noted and is used to calculate the corresponding audio frame number (AF) that has to be played at that time, given by (9).
(9)

The above equation is used to synchronize the audio and video streams. Once the frame numbers are obtained, the audio and video elementary streams can be constructed by writing the audio and video buffer contents from that point (frame) into their respective elementary streams i.e. .aac and .264 files respectively. Then the streams are merged into a container format by using mkv merge[31] which is a freely available software. The resulting container format can be played back by media players like VLC media player[32] or Gom media player[33]. In the case of video sequence, to ensure proper playback, picture parameter sets (PPS) and sequence parameter sets (SPS) must be inserted before the first IDR frame, because both PPS and SPS information are used by the decoder to find out the encoding parameters used.

The reason that the de-multiplexing is carried out from an IDR (instantaneous decoder refresh) frame is because the IDR frame breaks the video sequence making sure that the later frames like P or B-frames do not use frames before the IDR frame for motion estimation. This is not true in the case of normal I- frame. So in a long sequence, the GOPs after the IDR frame are treated as new sequences by the H.264 decoder. In the case of the audio HE AAC v2 decoder can playback the sequence from any audio frame.

Wednesday 23 December 2015

What are h264 stream formats? importance of NAL

H264 Stream Formats:

There are two H.264 stream formats and they are sometimes called
  1. Annex B
  2. MP4
An H.264 stream is made of NALs (a unit of packaging)
  1. Annex B: has start codes 0x00 0x00 0x00 0x01 NAL 0x00 0x00 0x00
    0x01 NAL etc
  2. MP4: is size prefixed SIZE NAL SIZE NAL etc
The MP4 stream format doesn't contain any NALs of type SPS, PPS or AU delimter.
The Annex B format you'll find in MPEG-2 TS, RTP and some encoders default output.
The MP4 format you'll find in MP4 files. Both formats can be converted into each other.
Annex B -> MP4: remove start codes, insert length of NAL, filter out SPS, PPS and AU delimiter.
MP4 -> Annex B: remove length, insert start code, insert SPS for each I-frame, insert PPS for each frame, insert AU delimiter for each GOP.

Monday 14 December 2015

Date/time missing in the menu bar in ubuntu

Date and time missing in menu bar in ubuntu? here is fix!

If you have upgraded your Ubuntu machine to Ubuntu Saucy 13.10, one of the things that you might find missing is the date/time indicator in the menu bar. And if you visit the “System Settings -> Time and Date -> Clock” section, you might find that the option to add it to the menu bar is greyed out and disabled.

 

If you are having this problem, here’s the fix.

1. Reinstall indicator-datetime. It should be installed by default, but just in case you have removed it unknowingly, it is best to run the install command again.
sudo apt-get install indicator-datetime
2. Next, we are going to reconfigure the date time:
sudo dpkg-reconfigure --frontend noninteractive tzdata
3. Lastly, restart unity.
sudo killall unity-panel-service
 NOTE: This was worked fine!! tested on ubuntu 14.04LTS.


















Wednesday 9 December 2015

Difference Between MOV and MP4

Which is the Best Choice between MOV and MP4 File Format

Today, there are different kinds of video file format supported across various platforms. The choice of a video file format totally depends upon the user requirement and the type of file he/she is working with. MOV and MP4 are the two most commonly used video file format. They both use the lossy video compression approach to hold the videos. This method makes the video much lighter in size as it eliminates the portion of the video that are less significant. In addition, it maintains a minimum quality loss after the video has been compressed.

MOV is a popular video format file originally developed by Apple. It was intended to support its QuickTime movie player and often used to save videos, movies, etc. It uses an advanced algorithm especially developed by Apple and is compatible across different versions of Mac and Windows operating systems.

It uses the concept of tracks to store data. A track, which stores a particular type of data, is stored in a multimedia container file. There are many such tracks present in the container. There are different tracks meant for holding different types of data like text, audio, video, etc. Such tracks maintain a hierarchy consisting of atoms as an object. In addition, it uses a specific format to contain a digitally encoded media stream. If the digitally encoded media stream is present in some other file, it makes a data reference to the media stream.

There are various advantages associated with MOV format, which makes it a much more popular and usable format for videos. However, it has some proprietary issues On the basis of the standard used by MOV format; MP4 file format was later developed. There were only minor modifications that too while data tagging information. Later, it was used as an industry standard.

Because of their almost identical nature, the MPEG-4 format can be used by both the MOV and MP4 container formats. In spite of the fact that MOV was intended for QuickTime player and MP4 uses the same lossy compression standards, they are mostly interchangeable in a QuickTime-only environment. A file in MOV format can be easily converted into MP4 and vice-versa without changing the video encoding. This is very much possible in case of Apple environment. If you are working in an environment other than Apple, you might face some complexities.

MP4 being an industry standard, has a larger support for operating systems other than Apple. There are various media players, which support this format in different operating systems. In addition, they are widely supported across various hand held devices like video players and gaming devices. In addition, it has a larger range of support for hardware devices like Sony PSP and various DVD players. On the software side, it includes most DirectShow / Video for Windows codec packs.

On the basis of MPEG-4 file format, the QuickTime (MOV) file format was approved by the International Organization for Standardization. In the year 2001, QuickTime format specification was published. On the basis of QuickTime format specification, the MPEG-4 file format specification was created. Later in the year 2001, the MPEG-4 specifications were revised on the basis of the specification published in 1999 and published the MP4 (.mp4) file format. With the advancement in technology and the requirements, it was in the year 2003, when the first version of MP4 format was revised. It was replaced by MPEG-4 Part 14.

MP4 can be used as a base of other multimedia files like 3GP, Motion JPEG 2000, etc. You can check the www.mp4ra.org website, which is the official registration authority website and contains all the registered extensions for ISO Base Media File Format.

To sum it up, MOV and MP4 are just containers and they do not present any real effect on the quality of the encoded videos. That is up to the codec like H.264 and the others. Choosing between MOV and MP4 should be based solely on where you want to play the resulting videos. If it is only meant to circulate around the Mac community, then you are pretty safe with MOV but if you want to put it in your PSP or any other non Apple portable device, then you are better off with MP4

Summary:

1.Both MOV and MP4 are lossy formats which sacrifices quality for file size
2.MOV was initially developed by Apple as a file format for QuickTime
3.MOV is the basis for the development of MP4
4.MP4 is the industry standard and has more widespread support than MOV
5.Saving your file to either MOV or MP4 would result in the same video if you use the same codec

 

Friday 20 November 2015

vim editor basics for beginners

VIM Editor is one of the familiar text editor in linux distributions.

Basics

vim my_file_name # open/create file with vim

Once you are in Vim the most important commands are i  :  and ESC. The i key brings you into the insert mode for typing. The ESC brings you out of there. And the : key starts the command mode at the bottom of the screen. In the following text, all commands starting with : need to be typed in the command mode. All other commands are typed in the normal mode after hitting the ESC key. 

Modifier Keys to Control Vim
  • i # INSERT MODE
  • ESC # NORMAL (NON-EDITING) MODE
  • : # commands start with ':'
  • :w # save command; if you are in editing mode you have to hit ESC first!!
  • :q # quit file, don't save
  • :q! # exits WITHOUT saving any changes you have made
  • :wq # save and quit
  • R # replace MODE
  • r # replace only one character under cursor
  • q: # history of commands (from NORMAL MODE!), to reexecute one of them, select and hit enter!
  • :w new_filename # saves into new file
  • :#,#w new_filename # saves specific lines (#,#) to new file
  • :# go to specified line number

Help

Help from Command Line

vimtutor # open vim tutorial from shell

Help in Vim
  • :help # opens help within vim, hit :q to get back to your file
  • :help <topic> # opens help on specified topic
  • :help_topic| CTRL-] # when you are in help this command opens help topic specified between |...|, CTRL-t brings you back to last topic
  • $ :help <topic> CTRL-D # gives list of help topics that contain key word
  • $ : <up-down keys> # like in shell you get recent commands!!!!

Moving Around in Files

  • $ # moves cursor to end of line
  • A # same as $, but switches to insert mode
  • 0 (zero) # moves cursor to beginning of line
  • CTRL-g # shows at status line filename and the line you are on
  • SHIFT-G # brings you to bottom of file, type line number (isn't displayed) then SHIFT-G # brings you to specified line#

Line Wrapping and Line Numbers

  • :set nowrap # no word wrapping, :set wrap # back to wrapping
  • :set number # shows line numbers, :set nonumber # back to no-number mode

Working with Many Files & Splitting Windows

vim -o *.txt # opens many files at once and displays them with horizontal 
             # split, '-O' does vertical split

vim *.txt    # opens many files at once; ':n' switches between files

  • :wall or :qall # write or quit all open files
  • :args *.txt # places all the relevant files in the argument list
  • :all # splits all files in the argument list (buffer) horizontally
  • CTRL-w # switch between windows
  • :split # shows same file in two windows
  • :split <file-to-open> # opens second file in new window
  • :vsplit # splits windows vertically, very useful for tables, ":set scrollbind" let's you scroll all open windows simultaneously
  • :close # closes current window
  • :only # closes all windows except current one

Spell Checking & Dictionary

  • :set spell # turns on spell checking
  • :set nospell # turns spell checking off
  • :! dict <word> # meaning of word
  • :! wn 'word' -over # synonyms of word

Enabling Syntax Highlighting

  • :set filetype=perl # Turns on syntax coloring for a chosen programming language.
  • :set syntax on # Turns syntax highlighting on
  • :set syntax off # Turns syntax highlighting off

Undo and Redo

  • u # undo last command
  • U # undo all changes on current line
  • CTRL-R # redo one change which was undone

Deleting Things

  • x # deletes what is under cursor
  • dw # deletes from curser to end of word including the space
  • de # deletes from curser to end of word NOT including the space
  • cw # deletes rest of word and lets you then insert, hit ESC to continue with NORMAL mode
  • c$ # deletes rest of line and lets you then insert, hit ESC to continue with with NORMAL mode
  • d$ # deletes from cursor to the end of the line
  • dd # deletes entire line
  • 2dd # deletes next two lines, continues: 3dd, 4dd and so on.

Copy & Paste

  • yy # copies line, for copying several lines do 2yy, 3yy and so on
  • p # pastes clipboard behind cursor

Search in Files

  • /my_pattern # searches for my_pattern downwards, type n for next match
  • ?my_pattern # seraches for my_pattern upwards, type n for next match
  • :set ic # switches to ignore case search (case insensitive)
  • :set hls # switches to highlight search (highlights search hits)

Replacements with Regular Expression Support

Great intro: A Tao of Regular Expressions
  • :s/old_pat/new_pat/ # replaces first occurrence in a line
  • :s/old_pat/new_pat/g # replaces all occurrence in a line
  • :s/old_pat/new_pat/gc # add 'c' to ask for confirmation
  • :#,#s/old_pat/new_pat/g # replaces all occurrence between line numbers: #,#
  • :%s/old_pat/new_pat/g # replaces all occurrence in file
  • :%s/\(pattern1\)\(pattern2\)/\1test\2/g # regular expression to insert, you need here '\' in front of parentheses (<# Perl)
  • :%s/\(pattern.*\)/\1 my_tag/g # appends something to line containing pattern (<# .+ from Perl is .* in VIM)
  • :%s/\(pattern\)\(.*\)/\1/g # removes everything in lines after pattern
  • :%s/\(At\dg\d\d\d\d\d\.\d\)\(.*\)/\1\t\2/g # inserts tabs between At1g12345.1 and Description
  • :%s/\n/new_pattern/g #Replaces return signs
  • :%s/pattern/\r/g #Replace pattern with return signs!!
  • :%s/\(\n\)/\1\1/g # insert additional return signs
  • :%s/\(^At\dg\d\d\d\d\d.\d\t.\{-}\t.\{-}\t.\{-}\t.\{-}\t\).\{-}\t/\1/g # replaces content between 5th and 6th tab (5th column), '{-}' turns off 'greedy' behavior
  • :#,#s/\( \{-} \|\.\|\n\)/\1/g # performs simple word count in specified range of text
  • :%s/\(E\{6,\}\)/<font color="green">\1<\/font>/g # highlight pattern in html colors, here highlighting of >= 6 occurences of Es
  • :%s/\([A-Z]\)/\l\1/g # change uppercase to lowercase, '%s/\([A-Z]\)/\u\1/g' does the opposite
  • :g/my_pattern/ s/\([A-Z]\)/\l\1/g | copy $ # uses 'global' command to apply replace function only on those lines that match a certain pattern. The 'copy $' command after the pipe '|' prints all matching lines at the end of the file.
  • :args *.txt | all | argdo %s/\old_pat/new_pat/ge | update # Command 'args' places all relevant files in the argument list (buffer); 'all' displays each file in separate split window; command 'argdo' applies replacement to all files in argument list (buffer); flag 'e' is necessary to avoid stop at error messages for files with no matches; command 'update' saves all changes to files that were updated.

Useful Utilities in Vim

Matching Parentheses
    • Place curser on (, [ or { and type % # curser moves to matching parentheses
Printing and Inserting Files
  • :ha # prints entire file
  • :#,#ha # prints specified lines: #,#
  • :r <filename> # inserts content of specified file after cursor

Convert Text File to HTML Format
  • :runtime! syntax/2html.vim # run this command with open file in Vim
Shell Commands in Vim
  • :!<SHELL_COMMAND> <ENTER> # executes any shell command, hit <enter> to return
  • :sh # switches window to shell, 'exit' switches back to vim
Using Vim as Table Editor
  • v # starts visual mode for selecting characters
  • V # starts visual mode for selecting lines
  • CTRL-V # starts visual mode for selecting blocks (use CTRL-q in gVim under Windows). This allows column-wise selections and operations like inserting and deleting columns. To restrict substitute commands to a column, one can select it and switch to the command-line by typing ':'. After this the substitution syntax for a selected block looks like this: '<,'>s///.
  • :set scrollbind # starts simultaneous scrolling of 'vsplitted' files. To set to horizontal binding of files, use command ':set scrollopt=hor' (after first one). Run all these commands before the ':split' command.
  • :AlignCtrl I= \t then :%Align # This allows to align tables by column separators (here '\t') when the Align utility from Charles Campbell's is installed. To sort table rows by selected lines or block, perform the visual select and then hit F3 key. The rest is interactive. To enable this function, one has to include in the .vimrc file the Vim sort script from Gerald Lai.

Modify Vim Settings 

The default settings in Vim are controlled by the .vimrc file in your home directory.
  •  see last chapter of vimtutor (start from shell)
  •  useful .vimrc sample
  •  when vim starts to respond very slowly then one may need to delete the .viminf* files in home directory

Linux Basics for Beginners

Logging-In

LINUX
  • To log-in into the remote Linux shell, open terminal and type:
    • ssh -X <your_username>@<host_name>
host name is the remote server's domain name (e.g. biocluster.ucr.edu)
You will be asked to enter the password, simply type it and press enter.
  • To copy files To the server run the following on your workstation or laptop:
scp -r <path_to_directory> <your_username>@<host_name>:
  • To copy files From the server run the following on your workstation or laptop:
scp -r <your_username>@<host_name>:<path_to_directory> .

Command-Line Syntax for this Manual

  • Remember the UNIX/LINUX command line is case sensitive!
  • All commands in this manual are printed in gray code boxes.
  • Commands given in red are considered more important for beginners than commands given in black.
  • The hash (pound) sign "#" indicates end of a command and the start of a comment.
  • The notation <...> refers to variables and file names that need to be specified by the user. The symbols < and > need to be excluded.


Orientation 

Viewing and changing the present working directory:

pwd               # Get full path of the present working directory (same as "echo $HOME")

ls                # Content of pwd
ls -l             # Similar as ls, but provides additional info on files and directories
ls -a             # Includes hidden files (.name) as well
ls -R             # Lists subdirectories recursively
ls -t             # Lists files in chronological order

cd <dir_name>     # Switches into specified directory.
cd                # Brings you to the highest level of your home directory.
cd ..             # Moves one directory up
cd ../../         # Moves two directories up (and so on)
cd -                            # Go back to you were previously (before the last directory change)

The tilde symbol (~) gets interpreted as the path to your home directory. This will happen anywhere on the command line: 

echo ~            # View the full (complete) path of your home
find ~            # List all your files (including everything in sub-directories)
ls ~              # List the top level files of your home directory
du -sch ~/*       # Calculate the file sizes in your home

Viewing file info, user, and host:

stat <file-name>  # Last modification time stamps, permissions, and size of a file

whoami            # Shows your user name (same as "echo $USER")
hostname          # Shows on which machine you are (same as "echo $HOSTNAME")


Files and directories


mkdir <dir_name>   # Creates specified directory
rmdir <dir_name>   # Removes empty directory
rm <file_name>     # Removes file name
rm -r <dir_name>   # Removes directory including its content, but asks for confirmation, 
                   # 'f' argument turns confirmation off

cp <name> <path>   # Copy file/directory as specified in path (-r to include content in directories)
mv <name1> <name2> # Renames directories or files
mv <name> <path>   # Moves file/directory as specified in path

Friday 28 August 2015

what are storage classes in c?

Storage Classes in C:


From C compiler’s point of view, a variable name identifies some physical location within the computer where the string of bits representing the variable’s value is stored. There are basically two kinds of locations in a computer where such a value may be kept— Memory and CPU registers. It is the variable’s storage class that determines in which of these two locations the value is stored.

a variable’s storage class tells us:
(a)   Where the variable would be stored.
(b)   What will be the initial value of the variable, if initial value is not specifically assigned.(i.e. the default initial value).
(c)  What is the scope of the variable; i.e. in which functions the value of the variable would be available.
(d)  What is the life of the variable; i.e. how long would the variable exist.

There are four storage classes in C:

1. Automatic storage class
2. Register storage class
3. Static storage class
4. External storage class
Let us examine these storage classes one by one.

1. Automatic storage class:


The features of a variable defined to have an automatic storage
class are as under:
 
Storage                              Memory.

Default initial value  − An unpredictable value, which is often called a garbage value.

Scope                              −   Local to the block in which the variable is defined.

Life                                  −  Till the control remains within the block in which the variable is defined.Following program shows how an automatic storage class variable is declared, and the fact that if the variable is not initialized it contains a garbage value.

main( )
{
   auto int i, j ;
   printf ( "\n%d %d", i, j ) ;
}

The output of the above program could be...
1211 221

where, 1211 and 221 are garbage values of i and j. When you run this program you may get different values, since garbage values  are unpredictable. So always make it a point that you initialize the automatic variables properly, otherwise you are likely to get unexpected results. Note that the keyword for this storage class is auto, and not automatic.


2. Register storage class:


The features of a variable defined to be of register storage class are as under:
 
Storage                          -     CPU registers. 

Default initial value    -     Garbage value.

Scope                             -     Local to the block in which the variable is defined.

Life                                -    Till the control remains within the block in which the variable is defined.

A value stored in a CPU register can always be accessed faster than the one that is stored in memory. Therefore, if a variable is used at many places in a program it is better to declare its storage class as register. A good example of frequently used variables is loop counters. We can name their storage class as register.
main( )
{
   register int i ;
   for ( i = 1 ; i <= 10 ; i++ )
     printf ( "\n%d", i ) ;
}
Here, even though we have declared the storage class of i as register, we cannot say for sure that the value of i would be stored in a CPU register. Why? Because the number of CPU registers are limited, and they may be busy doing some other task. What happens in such an event... the variable works as if its storage class
is auto.

NOTE: Not every type of variable can be stored in a CPU register.

For example, if the microprocessor has 16-bit registers then they cannot hold a float value or a double value, which require 4 and 8 bytes respectively. However, if you use the register storage class for a float or a double variable you won’t get any error messages. All that would happen is the compiler would treat the variables to be of auto storage class.

3. Static storage class:


The features of a variable defined to have a static storage class are as under: 

Storage                          −      Memory.

Default initial value     −      Zero. 

Scope                             −      Local to the block in which the variable is defined.

Life                              − Value of the variable persists between different function calls.
main( )
{
     increment( ) ;
     increment( ) ;
     increment( ) ;
}
increment( )
{
     static int i = 1 ;
     printf ( "%d\n", i ) ;
     i = i + 1 ;
}
output: 1 2 3

the function increment( ) gets called from main( ) thrice. Each time it increments the value of i and prints it. static makes the variable exist even if the function returns. so next time increment called it won't create new memory for variable i and it will use the existing memory of i (created in first call).

 4. External storage class


The features of a variable whose storage class has been defined as external are as follows:

Storage                                   
−     Memory.

Default initial value              −     Zero.

Scope                                       −     Global.

Life                   −     As long as the program’s execution doesn’t come to an end.

 
External variables differ from those we have already discussed in that their scope is global, not local. External variables are declared outside all functions, yet are available to all functions that care to use them. Here is an example to illustrate this fact.

int i ;

main( )
{
     printf ( "\ni = %d", i ) ;
     increment( ) ;
     increment( ) ;
     decrement( ) ;
     decrement( ) ;
}

increment( )
{
     i = i + 1 ;
     printf ( "\non incrementing i = %d", i ) ;
}

decrement( )
{
     i = i - 1 ;
     printf ( "\non decrementing i = %d", i ) ;
}

The output would be:
i = 0
on incrementing i = 1
on incrementing i = 2
on decrementing i = 1
on decrementing i = 0


Which to Use When???


Use static storage class only if you want the value of a variable to persist between different function calls.

− Use register storage class for only those variables that are being used very often in a program. Reason is, there are very few CPU registers at our disposal and many of them might be busy doing something else. Make careful utilization of the scarce resources. A typical application of register storage class is loop counters, which get used a number of times in a program.

− Use extern storage class for only those variables that are being used by almost all the functions in the program. This would avoid unnecessary passing of these variables as arguments when making a function call. Declaring all the variables as extern would amount to a lot of wastage of memory space because these variables would remain active throughout the life of the program.

− If you don’t have any of the express needs mentioned above, then use the auto storage class. In fact most of the times we end up using the auto variables, because often it so happens that once we have used the variables in a function we don’t mind loosing them.