limit reading & writing to file

Posted By: Reconnoiter

limit reading & writing to file - 09/26/17 13:35

Howdy,

I am trying to append some text from a very big data file to an other big data file. I am using file_asc_read and file_asc_write for this. The appending goes fine however it eventually stops at line ~13982. I think it reached a limit or such. When I append the text another time it ends at line ~27963 (about twice as long). Anyway to bypass this limit? Here is the code that does the copying:

Code:

var _sizeReadFile = file_length(_handleReadFile); //get size
while (_sizeReadFile > 0) {
		var _byte = file_asc_read (_handleReadFile); //read byte
		file_asc_write (_handleSave, _byte); //save byte
		_sizeReadFile -= 1;
	}

Thanks

Posted By: txesmi

Re: limit reading & writing to file - 09/26/17 18:38

Hi,
you can copy the content to a buffer and save it. Here goes an untested example:

Code:

long _size = 0;
char *_file = file_load ( "myfile.txt", NULL, &_size );
char *_fileNew = sys_malloc ( _size * 2 );
memcpy ( _fileNew, _file, _size );
memcpy ( _fileNew + _size, _file, _size );
file_save ( "mynewfile.txt", _fileNew, _size * 2 );
file_load ( NULL, _file, NULL );
sys_free ( _fileNew );

Posted By: Reconnoiter

Re: limit reading & writing to file - 09/27/17 11:50

Hey txesmi that does copy the whole data ty, but how do I append it to the original file? (it now overwrites, something like memcat would be nice if that exists :S)

Posted By: MasterQ32

Re: limit reading & writing to file - 09/27/17 12:03

use the stdio.h file API to access files. you can open a file with

Code:

FILE * f = fopen("filename", "a");

to append data to the end of the file.

also try to prevent single-byte reads and read as much data as possible in a single fread call

Posted By: Reconnoiter

Re: limit reading & writing to file - 09/27/17 15:02

Hi, tnx for that link & tip. I almost got it working now, there is still a small bug in the code with big data files; as in that it copies & appends everything perfectly but then where it should stop it doesn't and instead appends puts some extra crap. Also the error("reading error") triggers (see code below) so I am probably setting some size variable wrong.

Code:

FILE * _fWrite = fopen(_chr(tmpWrite_str), "a");
	if (_fWrite != NULL) {
		//		error("works");
		FILE * _fRead = fopen(_chr(tmpRead_str), "r");
		if (_fRead != NULL) {
			
			long _lSize;
			char* _buffer;
			size_t _result;
			
			// obtain file size
			fseek (_fRead, 0, SEEK_END);
			_lSize = ftell (_fRead);
			rewind (_fRead);

			// allocate memory to contain the whole file
			_buffer = (char*) malloc (sizeof(char)*_lSize);
			if (_buffer == NULL) error("memory error");

			// copy the file into the buffer
			_result = fread (_buffer, 1, _lSize, _fRead);
			if (_result != _lSize) error("reading error");
			
			// append buffer to write file
			fwrite (_buffer, 1, _lSize, _fWrite);
			
			fclose (_fRead);
			free (_buffer);
		}
		fclose (_fWrite);
	}
	//	else error("error");