Sponsored Content
Top Forums Shell Programming and Scripting Optimised way for search & replace a value on one line in a very huge file (File Size is 24 GB). Post 302552323 by yazu on Friday 2nd of September 2011 11:37:59 AM
Old 09-02-2011
Afaik, in the general case no. But if your new first line has the same number of bytes or shorter and you can pad it with spaces then I believe you can do it quick with low level programming - open for read/write, read some bytes (512 for example) in a buffer, change them, rewind, and write the buffer back.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

In Line File Modifications: Search and Replace

grep -il "TEST" ${ENVIRON}/*.pde| while read pde   do     cat $pde | sed s/"TEST 3,1"/"TEST 3,0"/g | sed s/"TEST  3,1"/"TEST  3,0"/g > ${pde}.tmp     if ; then       mv ${pde}.tmp $pde ... (2 Replies)
Discussion started by: Shakey21
2 Replies

2. UNIX for Dummies Questions & Answers

how can search a String in one text file and replace the whole line in another file

i am very new to UNIX plz help me in this scenario i have two text files as below file1.txt name=Rajakumar. Discipline=Electronics and communication. Designation=software Engineer. file2.txt name=Kannan. Discipline=Mechanical. Designation=CADD Design Engineer. ... (6 Replies)
Discussion started by: kkraja
6 Replies

3. UNIX for Dummies Questions & Answers

How to search and replace a particular line in file with sed command

Hello, I have a file and in that, I want to search for a aprticular word and then replace another word in the same line with something else. Example: In file abc.txt, there is a line <host oa_var="s_hostname">test</host> I want to search with s_hostname text and then replace test with... (2 Replies)
Discussion started by: sshah1001
2 Replies

4. Shell Programming and Scripting

Search & Replace in Multiple Files by reading a input file

Hi, I have a folder which contains multiple config.xml files and one input file, Please see the below format. Config Files format looks like :- Code: <application name="SAMPLE-ARCHIVE"> <NVPairs name="Global Variables"> <NameValuePair> ... (0 Replies)
Discussion started by: haiksuresh
0 Replies

5. Shell Programming and Scripting

Implement in one line sed or awk having no delimiter and file size is huge

I have file which contains around 5000 lines. The lines are fixed legth but having no delimiter.Each line line contains nearly 3000 characters. I want to delete the lines a> if it starts with 1 and if 576th postion is a digit i,e 0-9 or b> if it starts with 0 or 9(i,e header and footer) ... (4 Replies)
Discussion started by: millan
4 Replies

6. Shell Programming and Scripting

Global search and replace multi line file

Hello I need to search for a mult-line strngs(with spaces in between and qoted) in a file1 and replace that text with Fixed string globally in file1. The strng to search for is in file2. The file is big with some 20K records. so speed and effciency is required file1: (where srch & rplc... (0 Replies)
Discussion started by: Hiano
0 Replies

7. Shell Programming and Scripting

Mutli line pattern search & replace in a xml file

Hello guys, I need your help for a specific sed command that would search for a multi line pattern and if found, would replace it by another multi line pattern. For instance, here is the input: <RefNickName>abcd</RefNickName> <NickName>efgh</NickName> <Customize> ... (0 Replies)
Discussion started by: xciteddd
0 Replies

8. Shell Programming and Scripting

awk search/replace specific field, using variables for regexp & subsitution then overwrite file

Hello, I'm trying the solve the following problem. I have a file which I intend to use as a csv called master.csv The columns are separated by commas. I want to change the text on a specific row in either column 3,4,5 or 6 from xxx to yyy depending upon if column 1 matches a specified pattern.... (3 Replies)
Discussion started by: cyphex
3 Replies

9. Shell Programming and Scripting

Search & Replace in Multiple Files by reading a input file

I have a environment property file which contains: Input file: value1 = url1 value2 = url2 value3 = url3 and so on. I need to search all *.xml files under directory for value1 and replace it with url1. Same thing I have to do for all values mentioned in input file. I need script in unix bash... (7 Replies)
Discussion started by: Shamkamde
7 Replies

10. UNIX for Dummies Questions & Answers

Need to replace new line characters in a huge file

Hi , I would like to replace new line characters(\n) in a huge file of about 2 million records . I tried this one (:%s/\n//g) but it's hanging there and no result. Does this command do not work if the file is big. Please let me know if you have any other options Regards Raj (1 Reply)
Discussion started by: rajeevm
1 Replies
VBUF(1) 						      General Commands Manual							   VBUF(1)

NAME
vbuf - Virtual Ring Buffer shell interface SYNOPSIS
vbuf [options] DESCRIPTION
This manual page documents briefly the vbuf command. vbuf is a program to glue two programs (a producer and a consumer) together, one of which not being able to fulfil the other's constant data rate requirements. OPTIONS
-h Show short help message. --help Show long help message. -p, --progress Show progress status line. -q, --quiet Suppress progress status line. The default is to show the progress display. -s size, --size=size Set buffer size in bytes. Suffixes k, m, and g may be used. The actual size used will be rounded up for system mapping require- ments. The default is 1m, or 1048576 bytes. -t time, --time=seconds Set time in seconds between progress display refreshes. Fractions of a second may be used. Default is 1.500000 seconds. -b, --bits Set progress rate display units to bits per second. -B, --bytes Set progress rate display units to bytes per second. The default is bytes. -d, --decimal Set progress rate display base to decimal. The default is decimal. -x, --hexadecimal Set progress rate display base to hexadecimal. --octal Set progress rate display base to octal. -i file, --input=file Specify an input file to be opened and used instead of using stdin. -o file, --output=file Specify an output file to be opened and used instead of using stdout. -M file, --mapfile=file Specify a file which will be used for backing store for the virtual ring buffer. The allows using filesystem space in case swap space is full. -r bytes, --read-min=bytes Specify the minimum number of bytes to be requested by read. Reading will not happen unless at least this much buffer space is available. The default is 1. -R bytes, --read-max=bytes Specify the maximum number of bytes to be requested by read. Reading will not request more even if more buffer space is available. The default is the full buffer size. -w bytes, --write-min=bytes Specify the minumum number of bytes to be written. Writing will not be done unless at least this much data is available. Once end of file is reached on input, a smaller amount may be written. The default is 1. -W bytes, --write-max=bytes Specify the maximum number of bytes to be written. Writing will not write more each time even if more data is available in the buf- fer. The default is the full buffer size. AUTHOR
VRB and vbuf were written by Phil Howard <vrb@ipal.org>. This manual page was written by Szekelyi Szabolcs <cc@mail.3d.hu>, for the Debian project (but may be used by others). 2006 Sep 5 VBUF(1)
All times are GMT -4. The time now is 07:25 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy