Sweep: Implement uploading of files for model fine-tuning to OpenAI API

[lehcode]

Checklist

• src/SoakpProxy.ts

• Add a new method uploadFineTuneFile that takes a file and a purpose as parameters. This method should use the OpenAI API's createFile method to upload the file. The purpose should be 'fine-tune'.

• src/openai/OpenaiFinetunesApi.ts

• Add a new endpoint POST /openai/fine-tunes/upload that handles the HTTP request for file upload. This endpoint should call the uploadFineTuneFile method in the SoakpProxy class to upload the file to the OpenAI API.
• Convert uploaded files into JSONL with Python means.

Checklist

• src/SoakpProxy.ts

• Add a new method uploadFineTuneFile that takes a file and a purpose as parameters. This method should use the OpenAI API's createFile method to upload the file. The purpose should be 'fine-tune'.

• src/openai/OpenaiFinetunesApi.ts

• Add a new endpoint POST /openai/fine-tunes/upload that handles the HTTP request for file upload. This endpoint should call the uploadFineTuneFile method in the SoakpProxy class to upload the file to the OpenAI API.

• src/openai/OpenaiFilesApi.ts

• Add a new method convertToJSONL that takes a file as a parameter and converts it into JSONL format using Python means. This method should be called before the file is uploaded to the OpenAI API.

[sweep-ai-deprecated]

Here's the PR! #162.

⚡ Sweep Free Trial: I used GPT-3.5 to create this ticket. You have 4 GPT-4 tickets left for the month and 0 for the day. For more GPT-4 tickets, visit our payment portal. To retrigger Sweep edit the issue.

---

Step 1: 🔍 Code Search

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I looked at (click to expand). If some file is missing from here, you can mention the path in the ticket description.

soakp/src/openai/OpenaiFinetunesApi.ts

Lines 1 to 213 in 2ff492c

	import { SoakpServer } from '../SoakpServer';
	import validateToken from '../middleware/validateToken';
	import getProxyInstance from '../middleware/getProxyInstance';
	import { SoakpProxy } from '../SoakpProxy';
	import express from 'express';
	import { StatusCode } from '../enums/StatusCode.enum';
	import { StatusMessage } from '../enums/StatusMessage.enum';
	import { Responses } from '../lib/Responses';
	import { CreateFineTuneRequest } from 'openai';
	import { Project } from 'ts-morph';
	/**
	* `OpenaiFinetunesApi` class provides a set of methods that work with the OpenAI API to handle fine-tuning related jobs.
	* The methods provided allow to create, list, retrieve details, list events, and cancel fine-tuning jobs.
	* Each method corresponds to an API endpoint, and uses an Express application service and a proxy instance to communicate with OpenAI.
	* This class must be instantiated with a `SoakpServer` context which provides access to application services and utilities.
	*/
	export class OpenaiFinetunesApi {
	/**
	* Express application
	*
	* @private
	*/
	private appService: express.Application;
	/**
	* Proxy instance
	*
	* @private
	*/
	private proxy: SoakpProxy;
	/**
	* OpenaiFinetunesApi constructor
	*
	* @constructor
	* @param ctx
	*/
	constructor(ctx: SoakpServer) {
	this.appService = ctx.getApp();
	this.proxy = ctx.proxy;
	this.appService.post('/openai/fine-tunes',
	validateToken(ctx.jwtHash, ctx.getKeyStorage(), ctx.getUser()),
	getProxyInstance(ctx),
	this.createJob.bind(ctx));
	this.appService.get('/openai/fine-tunes',
	validateToken(ctx.jwtHash, ctx.getKeyStorage(), ctx.getUser()),
	getProxyInstance(ctx),
	this.listFineTunes.bind(ctx));
	this.appService.get('/openai/fine-tunes/:fine_tune_id',
	validateToken(ctx.jwtHash, ctx.getKeyStorage(), ctx.getUser()),
	getProxyInstance(ctx),
	this.getJob.bind(ctx));
	this.appService.get('/openai/fine-tunes/:fine_tune_id/events',
	validateToken(ctx.jwtHash, ctx.getKeyStorage(), ctx.getUser()),
	getProxyInstance(ctx),
	this.listEvents.bind(ctx));
	this.appService.post('/openai/fine-tunes/:fine_tune_id/cancel',
	validateToken(ctx.jwtHash, ctx.getKeyStorage(), ctx.getUser()),
	getProxyInstance(ctx),
	this.cancelJob.bind(ctx));
	}
	/**
	* `createJob` method initiates a fine-tuning job on the specified model using a given dataset.
	* It expects the incoming request body to include `training_file` (ID of the file to be used for training)
	* and `model` (model to be fine-tuned) parameters. It communicates with the OpenAI API via SoakpProxy.
	*
	* @param req
	* @param res
	*
	* TODO: Add input validation
	*/
	async createJob(req: express.Request, res: express.Response) {
	try {
	const fileId = String(req.body.training_file);
	const response = await this.proxy.createFineTune(fileId);
	if (response.status === StatusCode.SUCCESS) {
	Responses.success( res, { response: response.data, responseConfig: response.config.data }, StatusMessage.RECEIVED_OPENAI_API_RESPONSE );
	}
	// @ts-ignore
	} catch (err: Error) {
	if (err.response.status === StatusCode.BAD_REQUEST) {
	Responses.error( res, err.message, StatusCode.BAD_REQUEST );
	} else {
	Responses.error( res, err.message, StatusCode.INTERNAL_ERROR );
	}
	}
	}
	/**
	* `listFineTunes` method retrieves a list of all fine-tuning jobs of the organization.
	* It does not require any specific parameters and fetches the data from OpenAI API via SoakpProxy.
	*
	* @param req
	* @param res
	*/
	async listFineTunes(req: express.Request, res: express.Response) {
	try {
	const response = await this.proxy.listFineTunes();
	if (response.status === StatusCode.SUCCESS) {
	Responses.success( res, { response: response.data, responseConfig: response.config.data }, StatusMessage.RECEIVED_OPENAI_API_RESPONSE );
	}
	// @ts-ignore
	} catch (err: Error) {
	Responses.error( res, err.message, StatusCode.INTERNAL_ERROR );
	}
	}
	/**
	* `getJob` method retrieves detailed information about a specific fine-tuning job.
	* It expects a `fine_tune_id` parameter in the request URL that specifies the ID of the fine-tuning job to fetch.
	* The method communicates with the OpenAI API via proxy.
	*
	* @param req
	* @param res
	*/
	async getJob(req: express.Request, res: express.Response) {
	try {
	const jobId = String(req.params.fine_tune_id);
	const response = await this.proxy.getFineTuneJob(jobId);
	if (response.status === StatusCode.SUCCESS) {
	Responses.success( res, { response: response.data, responseConfig: response.config.data }, StatusMessage.RECEIVED_OPENAI_API_RESPONSE );
	}
	// @ts-ignore
	} catch (err: Error) {
	Responses.error( res, err.message, StatusCode.INTERNAL_ERROR );
	}
	}
	/**
	* `listEvents` method retrieves a list of all events related to a specific fine-tuning job.
	* It expects a `fine_tune_id` parameter in the request URL that specifies the ID of the fine-tuning job.
	* The method communicates with the OpenAI API via proxy.
	*
	* @param req
	* @param res
	*/
	async listEvents(req: express.Request, res: express.Response) {
	try {
	const jobId = String(req.params.fine_tune_id);
	const response = await this.proxy.getFineTuneJobEvents(jobId);
	if (response.status === StatusCode.SUCCESS) {
	Responses.success( res, { response: response.data, responseConfig: response.config.data }, StatusMessage.RECEIVED_OPENAI_API_RESPONSE );
	}
	// @ts-ignore
	} catch (err: Error) {
	Responses.error( res, err.message, StatusCode.INTERNAL_ERROR );
	}
	}
	/**
	* `cancelJob` method cancels a currently running fine-tuning job.
	* It expects a `fine_tune_id` parameter in the request URL that specifies the ID of the fine-tuning job to be cancelled.
	* The method communicates with the OpenAI API via proxy.
	*
	* @param req
	* @param res
	*/
	async cancelJob(req: express.Request, res: express.Response) {
	try {
	const jobId = String(req.params.fine_tune_id);
	const response = await this.proxy.cancelFineTuneJob(jobId);
	if (response.status === StatusCode.SUCCESS) {
	Responses.success( res, { response: response.data, responseConfig: response.config.data }, StatusMessage.RECEIVED_OPENAI_API_RESPONSE );
	}
	// @ts-ignore
	} catch (err: Error) {
	Responses.error( res, err.message, StatusCode.INTERNAL_ERROR );
	}
	}
	/**
	* `uploadFineTuneFile` method uploads a file for fine-tuning a model.
	* It expects a file and a purpose in the request body. The purpose should be 'fine-tune'.
	* The method communicates with the OpenAI API via proxy.
	*
	* @param req
	* @param res
	*/
	async uploadFineTuneFile(req: express.Request, res: express.Response) {
	try {
	const file = req.body.file;
	const purpose = req.body.purpose || 'fine-tune';
	if (!file) {
	Responses.error(res, 'File is required.', StatusCode.BAD_REQUEST);
	return;
	}
	if (!this.proxy) {
	Responses.error(res, 'Invalid proxy object.', StatusCode.INTERNAL_ERROR);
	return;
	}
	const response = await this.proxy.uploadFile(file, purpose);
	if (response.status === StatusCode.SUCCESS) {
	Responses.success(res, { response: response.data, responseConfig: response.config.data }, StatusMessage.RECEIVED_OPENAI_API_RESPONSE);
	} else {
	Responses.error(res, 'An error occurred during file upload.', StatusCode.INTERNAL_ERROR);
	}
	} catch (err: any) {
	Responses.error(res, err.message, StatusCode.INTERNAL_ERROR);
	}
	}
	}

soakp/src/SoakpProxy.ts

Lines 1 to 397 in 2ff492c

	/**
	* Author: Lehcode
	* Copyright: (C)2023
	*/
	import { Configuration, CreateChatCompletionRequest, CreateFineTuneRequest, OpenAIApi } from 'openai';
	import fs, { promises } from 'fs';
	import path from 'path';
	import { serverConfig } from './configs';
	import { StatusMessage } from './enums/StatusMessage.enum';
	import Stream, { Readable } from 'stream';
	import jsonlines from 'jsonlines';
	import readline from 'readline';
	import { LineParser } from './lib/LineParser';
	import { mergeMap, toArray } from 'rxjs/operators';
	import { from, Observable, bindNodeCallback } from 'rxjs';
	import { AxiosPromise } from 'axios';
	import { FineTune } from 'openai/api';
	/**
	* @class SoakpProxy
	*/
	export class SoakpProxy {
	/**
	* OpenAI API
	*
	* @private
	*/
	private openai: OpenAIApi;
	/**
	* @constructor
	*/
	constructor() {
	//
	}
	/**
	*
	* @param config
	*/
	initOpenai(config: Configuration) {
	const configuration = new Configuration({
	apiKey: config.apiKey || '',
	organization: config.organization || null
	});
	this.openai = new OpenAIApi(configuration);
	if (process.env.NODE_ENV === 'production') {
	console.log(`${StatusMessage.INITIALIZED_SOAKP_PROXY_WITH_API_KEY} '[scrubbed]'`);
	} else {
	// const sub = config.apiKey.substring(0, Math.round(config.apiKey.length/2));
	console.log(`${StatusMessage.INITIALIZED_SOAKP_PROXY_WITH_API_KEY} '[scrubbed]'`);
	}
	}
	/**
	*
	* Make OpenAI API call
	*/
	async chatRequest(request: CreateChatCompletionRequest) {
	try {
	return await this.openai.createChatCompletion(request);
	} catch (error) {
	throw error;
	}
	}
	/**
	* Get list of OpenAI models with properties
	*/
	async listModels() {
	return await this.openai.listModels();
	}
	/**
	* Upload a file that contains document(s) to be used across various
	* endpoints/features. Currently, the size of all the files uploaded by one
	* organization can be up to 1 GB.
	* Please contact us if you need to increase the storage limit.
	* Purpose can be 'answers' or 'fine-tune'
	*
	* @param file
	* @param purpose
	*/
	async uploadFile(file: any, purpose?: string) {
	try {
	return await this.openai.createFile(file, purpose);
	} catch (err: any) {
	console.log(err);
	if (err instanceof TypeError) {
	throw new Error(err.message);
	}
	}
	}
	/**
	*
	* @param txtFile
	* @param completion
	*/
	async txt2Jsonlines(txtFile: Record<string, any>, completion: string): Promise<Record<string, string>> {
	return new Promise((resolve, reject) => {
	try {
	const buffer = Buffer.from(txtFile.buffer);
	const basename = path.basename(txtFile.originalname);
	const readableStream = new Stream.Readable();
	readableStream.push(buffer);
	readableStream.push(null);
	const jsonlFilePath = path.resolve(`${serverConfig.dataDir}/jsonl/${basename}.jsonl`);
	const writeStream = fs.createWriteStream(jsonlFilePath);
	const stringify = jsonlines.stringify();
	const readStream = readline.createInterface({
	input: readableStream,
	output: process.stdout,
	terminal: false
	});
	stringify.pipe(process.stdout);
	stringify.pipe(writeStream);
	let lineNumber = 0;
	readStream.on('line', (line) => {
	lineNumber++;
	if (line !== '') {
	const cleanLine = line.replace(/^\s+/, '');
	// Convert the line to a JSON object
	stringify.write({ prompt: `${cleanLine}\\n\\n###\\n\\n`, completion: ` ${completion} END` }, (err) => {
	if (err) {
	reject(err);
	}
	});
	}
	});
	readStream.on('close', () => {
	// Signal the end of the stringify stream
	stringify.end();
	});
	writeStream.on('finish', () => {
	console.log('Done converting buffer to .jsonl');
	new Promise(async () => {
	const jsonlData = await fs.promises.readFile(jsonlFilePath, 'utf8');
	resolve({
	file: jsonlFilePath,
	data: jsonlData
	});
	});
	});
	} catch (err: any) {
	reject(err);
	}
	});
	}
	/**
	* Concatenate array of files into single JSONL file suitable for OpenAI mdodel fine-tuning
	*
	* @param txtFiles
	* @param completions
	* @param concatBaseName?
	*/
	async concatTxt2Jsonlines(
	txtFiles: Express.Multer.File[] | { [p: string]: Express.Multer.File[] },
	completions: string[],
	concatBaseName?: string
	) {
	return new Promise<Express.Multer.File>((resolve, reject) => {
	try {
	const jsonlFileName = concatBaseName || `concatenated-${Date.now()}.jsonl`;
	const jsonlFilePath = path.resolve(
	`${serverConfig.dataDir}/jsonl/${jsonlFileName}.jsonl`
	);
	const writeStream = fs.createWriteStream(jsonlFilePath);
	const stringify = jsonlines.stringify();
	stringify.pipe(writeStream);
	const txtFilesArray = Array.isArray(txtFiles) ? txtFiles : Object.values(txtFiles);
	from(txtFilesArray).pipe(
	// @ts-ignore
	mergeMap((txtFile: Express.Multer.File, index: number) => {
	const buffer = Buffer.from(txtFile.buffer);
	const readableStream = new Readable();
	readableStream.push(buffer);
	readableStream.push(null);
	return this.readableStreamToObservable(readableStream).pipe(
	mergeMap((lineBuffer: Buffer) => {
	const line = lineBuffer.toString('utf8');
	if (line.trim() !== '') {
	const cleanedLine = LineParser.cleanup(line);
	return bindNodeCallback(stringify.write.bind(stringify))({
	prompt: `${cleanedLine}\n\n###\n\n`,
	completion: ` ${completions[index]} END`
	});
	}
	})
	);
	})
	)
	.subscribe({
	complete: () => {
	stringify.end();
	},
	error: (err) => {
	console.log(err);
	reject(err);
	}
	});
	writeStream.on('finish', async () => {
	console.log(`Done concatenating files to ${jsonlFileName}`);
	try {
	const jsonlData = await fs.promises.readFile(jsonlFilePath);
	// @ts-ignore
	const concatenatedFile: Express.Multer.File = {
	...txtFilesArray[0],
	mimetype: 'application/json',
	originalname: jsonlFileName,
	buffer: jsonlData,
	path: jsonlFilePath
	};
	resolve(concatenatedFile);
	} catch (err) {
	reject(err);
	}
	});
	} catch (err: any) {
	reject(err);
	}
	});
	}
	/**
	*
	* @param readStream
	*/
	readableStreamToObservable(readStream: NodeJS.ReadableStream): Observable<Buffer> {
	return from(readStream).pipe(
	mergeMap((chunk: Buffer | string | null) => {
	if (chunk === null) {
	// Signal the end of the stream
	return;
	} else if (Buffer.isBuffer(chunk)) {
	return [chunk];
	} else if (typeof chunk === 'string') {
	return [Buffer.from(chunk, 'utf8')];
	} else {
	throw new Error('Unsupported chunk type.');
	}
	})
	);
	}
	/**
	* Parse JSON file to JSON object
	*
	* @param jsonl
	*/
	async parseJSONL(jsonl: string): Promise<any[]> {
	return new Promise((resolve, reject) => {
	try {
	// const jsonlParser = jsonlines.parse();
	const dataObj: any[] = [];
	// Split the jsonl string into lines
	const lines = jsonl.split('\n');
	lines.forEach((line) => {
	// Parse each line to a JavaScript object and add it to the dataObj array
	try {
	const data = JSON.parse(line);
	dataObj.push(data);
	} catch (err) {
	console.log('Invalid JSON line:', line);
	}
	});
	resolve(dataObj);
	} catch (err) {
	console.log(`Error parsing JSONL: ${err}`);
	reject(err);
	}
	});
	}
	/**
	* Delete file from OpenAI storage
	*
	* @param fileId
	*/
	async deleteFile(fileId: string) {
	return await this.openai.deleteFile(fileId);
	}
	/**
	* Get single model information by ID
	*
	* @param modelId
	*/
	async getModel(modelId: string) {
	return await this.openai.retrieveModel(modelId);
	}
	/**
	* Get single file information by ID
	*
	* @param fileId
	*/
	async getFileInfo(fileId: string) {
	return await this.openai.retrieveFile(fileId);
	}
	/**
	* Download file by it's ID from OpenAI storage
	*
	* @param fileId
	*/
	async getFileData(fileId: string) {
	return await this.openai.downloadFile(fileId);
	}
	/**
	* Create model fine-tune job
	*
	* @param fileId
	*/
	async createFineTune(fileId: string) {
	return await this.openai.createFineTune({
	/**
	* The ID of an uploaded file that contains training data.
	*
	* @type {string}
	*/
	training_file: fileId,
	/**
	* The name of the base model to fine-tune.
	* You can select one of "ada", "babbage","curie", "davinci",
	* or a fine-tuned model created after 2022-04-21.
	*
	* @type {string}
	*/
	model: 'davinci'
	});
	}
	/**
	* List your organization's fine-tuning jobs
	*/
	async listFineTunes() {
	return await this.openai.listFineTunes();
	}
	/**
	* Retrieve fine-tune. Gets full info about the fine-tune job by its ID.
	*
	* @param fineTuneJobId
	*/
	async getFineTuneJob(fineTuneJobId: string) {
	return await this.openai.retrieveFineTune(fineTuneJobId);
	}
	/**
	* List fine-tune events. Get fine-grained status updates for a fine-tune job.
	*
	* @param jobId
	*/
	async getFineTuneJobEvents(jobId: string) {
	return await this.openai.listFineTuneEvents(jobId);
	}
	/**
	* Delete fine-tune model. Delete a fine-tuned model. You must have the Owner role in your organization.
	*
	* @param modelId
	*/
	async deleteFineTuneModel(modelId: string) {
	return await this.openai.deleteModel(modelId);
	}
	/**
	* Cancel fine-tune. Immediately cancel a fine-tune job.
	*
	* @param jobId
	*/
	async cancelFineTuneJob(jobId: string) {
	return await this.openai.cancelFineTune(jobId);
	}
	}

soakp/src/openai/OpenaiFilesApi.ts

Lines 1 to 187 in 2ff492c

	import express from 'express';
	import { Responses } from '../lib/Responses';
	import { StatusCode } from '../enums/StatusCode.enum';
	import { StatusMessage } from '../enums/StatusMessage.enum';
	import path from 'path';
	import { serverConfig } from '../configs';
	import fs, { createReadStream, PathLike } from 'fs';
	import { ServerConfigInterface, SoakpServer } from '../SoakpServer';
	import validateToken from '../middleware/validateToken';
	import getProxyInstance from '../middleware/getProxyInstance';
	import uploadFiles from '../middleware/uploadFiles';
	import extractFileId from '../middleware/extractFileId';
	import { SoakpProxy } from '../SoakpProxy';
	import { Readable, Stream, Duplex } from 'stream';
	import { from, Observer } from 'rxjs';
	import { mergeMap } from 'rxjs/operators';
	import { Timer } from '../lib/Timer';
	export class OpenaiFilesApi {
	/**
	* Express application
	*
	* @private
	*/
	private appService: express.Application;
	/**
	* Proxy instance
	*
	* @private
	*/
	private proxy: SoakpProxy;
	/**
	* Server configuration
	*
	* @protected
	*/
	protected config: ServerConfigInterface;
	/**
	* OpenaiFilesApi
	*
	* @constructor
	* @param ctx
	*/
	constructor(ctx: SoakpServer) {
	this.appService = ctx.getApp();
	this.proxy = ctx.proxy;
	this.config = ctx.config;
	this.appService.post(
	'/openai/files',
	validateToken(ctx.jwtHash, ctx.getKeyStorage(), ctx.getUser()),
	getProxyInstance(ctx),
	uploadFiles(),
	this.sendFiles.bind(ctx)
	);
	this.appService.get(
	'/openai/files',
	validateToken(ctx.jwtHash, ctx.getKeyStorage(), ctx.getUser()),
	getProxyInstance(ctx),
	this.listFiles.bind(ctx)
	);
	this.appService.delete(
	'/openai/files/:file_id',
	validateToken(ctx.jwtHash, ctx.getKeyStorage(), ctx.getUser()),
	getProxyInstance(ctx),
	this.deleteFile.bind(ctx)
	);
	this.appService.get(
	'/openai/files/:file_id',
	validateToken(ctx.jwtHash, ctx.getKeyStorage(), ctx.getUser()),
	getProxyInstance(ctx),
	extractFileId(),
	this.getFile.bind(ctx)
	);
	this.appService.get(
	'/openai/files/:file_id/content',
	validateToken(ctx.jwtHash, ctx.getKeyStorage(), ctx.getUser()),
	getProxyInstance(ctx),
	extractFileId(),
	this.getFileData.bind(ctx)
	);
	}
	/**
	* Send uploaded file to OpenAI API
	*
	* @param req
	* @param res
	*/
	private async sendFile(req: express.Request, res: express.Response) {
	try {
	const title = String(req.body.title);
	if (title === '') {
	return Responses.error(res, 'Invalid document title', StatusCode.BAD_REQUEST, StatusMessage.BAD_REQUEST);
	}
	req.body.convert = req.body.convert === 'true' || false;
	const docName = String(req.body.document_title);
	const purpose = 'fine-tune';
	const ext = path.extname(req.file.originalname);
	let response;
	if (ext === '.jsonl') {
	const file = `${serverConfig.dataDir}/jsonl/${req.file.filename}.jsonl`;
	await fs.promises.writeFile(file, req.file.buffer);
	// @ts-ignore
	response = await this.proxy.uploadFile(fs.createReadStream(file), purpose);
	} else {
	if (req.body.convert === true) {
	// Code to handle conversion if `convert` input field exists and is `true`
	const validExtensions = this.config.validFiles;
	if (validExtensions.test(req.file.originalname)) {
	// @ts-ignore
	const jsonlFile = await this.proxy.txt2jsonl(req.file, docName);
	// @ts-ignore
	const formFile = fs.createReadStream(jsonlFile);
	response = await this.proxy.uploadFile(formFile, purpose);
	}
	} else {
	return Responses.error(
	res,
	'Provided file is not JSONL and `convert` was not specified.',
	StatusCode.UNSUPPORTED_MEDIA_TYPE,
	StatusMessage.WRONG_FILE_TYPE
	);
	}
	}
	if (response.status === StatusCode.SUCCESS) {
	return Responses.success(
	res,
	{ response: response.data, responseConfig: response.config.data },
	StatusMessage.RECEIVED_OPENAI_API_RESPONSE
	);
	}
	} catch (error) {
	console.error(error);
	return Responses.serverError(res);
	}
	}
	/**
	* Process uploaded files. Create JSONL file from uploaded files and upload it to OpenAI API.
	*
	* @param req
	* @param res
	*/
	protected async sendFiles(req: express.Request, res: express.Response): Promise<void> {
	try {
	if (!req.files || !(req.files instanceof Array)) {
	Responses.error(res, 'File(s) not specified.', StatusCode.INTERNAL_ERROR, StatusMessage.UPLOAD_ERROR);
	}
	const files: Express.Multer.File[] = req.files as Express.Multer.File[];
	// Check titles
	if (req.files.length === 1 && !req.body.titles) {
	Responses.error(
	res,
	'File needs a title which was not specified.',
	StatusCode.INTERNAL_ERROR,
	StatusMessage.UPLOAD_ERROR
	);
	return;
	} else if (parseInt(String(req.files.length)) > 1 && !(req.body.titles instanceof Array)) {
	Responses.error(
	res,
	'Each file needs its corresponding title.',
	StatusCode.INTERNAL_ERROR,
	StatusMessage.UPLOAD_ERROR
	);
	return;
	}
	if (files.map((file: Express.Multer.File) => this.config.validFiles.test(file.originalname)).length !== parseInt(String(req.files.length))) {
	Responses.error(
	res,
	'Mixing of JSONL and non-JSONL files not allowed.',
	StatusCode.INTERNAL_ERROR,
	StatusMessage.UPLOAD_ERROR
	);
	return;

soakp/src/SoakpServer.ts

Lines 10 to 133 in 2ff492c

	import { StatusMessage } from './enums/StatusMessage.enum';
	import { SoakpProxy } from './SoakpProxy';
	import { appConfig } from './configs';
	import { Responses } from './lib/Responses';
	import { DbSchemaInterface, KeyStorage } from './KeyStorage';
	import https from 'https';
	import path from 'path';
	import fs from 'fs';
	import { OpenAIConfigInterface } from './interfaces/OpenAIConfig.interface';
	import { OpenaiChatApi } from './openai/OpenaiChatApi';
	import { OpenaiModelsApi } from './openai/OpenaiModelsApi';
	import { OpenaiFilesApi } from './openai/OpenaiFilesApi';
	import { StatusCode } from './enums/StatusCode.enum';
	import { UserInterface } from './interfaces/User.interface';
	import { OpenaiFinetunesApi } from './openai/OpenaiFinetunesApi';
	import rateLimit from 'express-rate-limit';
	export interface ServerConfigInterface {
	httpPort: number;
	sslPort: number;
	httpAuthUser: string;
	httpAuthPass: string;
	dataDir: string;
	openAI: OpenAIConfigInterface;
	validFiles: RegExp;
	}
	/**
	* @class SoakpServer
	*/
	export class SoakpServer {
	protected appService: Express;
	private readonly keyStorageService: KeyStorage;
	readonly config: ServerConfigInterface;
	private readonly chat: OpenaiChatApi;
	private readonly models: OpenaiModelsApi;
	private readonly files: OpenaiFilesApi;
	private user: UserInterface;
	proxy: SoakpProxy;
	private readonly finetunes: OpenaiFinetunesApi;
	private rateLimit: any;
	/**
	*
	* @param configuration
	* @param storage
	*/
	constructor(configuration: ServerConfigInterface, storage: KeyStorage) {
	this.config = { ...configuration };
	this.keyStorageService = storage;
	this.user = { token: undefined, apiKey: undefined, orgId: undefined };
	this.proxy = new SoakpProxy();
	this.rateLimit = rateLimit({
	// 1 minute
	windowMs: 60*1000,
	max: 5
	});
	this.initializeExpressApp();
	this.chat = new OpenaiChatApi(this);
	this.models = new OpenaiModelsApi(this);
	this.files = new OpenaiFilesApi(this);
	this.finetunes = new OpenaiFinetunesApi(this);
	console.log(this.config);
	try {
	if (this.basicAuthCredentialsValid()) {
	this.appService.post(
	'/jwt/generate',
	basicAuth({ users: { [this.config.httpAuthUser]: this.config.httpAuthPass }}),
	this.generateJwt.bind(this)
	);
	}
	} catch (err: any) {
	if (err instanceof Error) {
	throw new Error(err.message);
	}
	}
	}
	/**
	* Initialize Express App
	*
	* @private
	*/
	private initializeExpressApp() {
	this.appService = express();
	// Configure middleware
	this.appService.use(cors());
	this.appService.use(express.json());
	this.appService.use(express.urlencoded({ extended: true }));
	this.appService.use(this.rateLimit);
	}
	/**
	* Generate JWT secret
	*
	* @private
	*/
	private get secret(): string {
	return String(process.env.JWT_SECRET).trim();
	}
	/**
	* Sign JWT with encoded secret key
	*
	* @private
	*/
	get jwtHash(): string {
	return createHash('sha256')
	.update(this.secret)
	.digest('hex');
	}
	/**
	* Handle GET `/get-jwt` request.
	* Generate JWT token for user-provided OpenAI API key.
	*

soakp/src/openai/OpenaiFilesApi.ts

Lines 154 to 298 in 2ff492c

	try {
	if (!req.files || !(req.files instanceof Array)) {
	Responses.error(res, 'File(s) not specified.', StatusCode.INTERNAL_ERROR, StatusMessage.UPLOAD_ERROR);
	}
	const files: Express.Multer.File[] = req.files as Express.Multer.File[];
	// Check titles
	if (req.files.length === 1 && !req.body.titles) {
	Responses.error(
	res,
	'File needs a title which was not specified.',
	StatusCode.INTERNAL_ERROR,
	StatusMessage.UPLOAD_ERROR
	);
	return;
	} else if (parseInt(String(req.files.length)) > 1 && !(req.body.titles instanceof Array)) {
	Responses.error(
	res,
	'Each file needs its corresponding title.',
	StatusCode.INTERNAL_ERROR,
	StatusMessage.UPLOAD_ERROR
	);
	return;
	}
	if (files.map((file: Express.Multer.File) => this.config.validFiles.test(file.originalname)).length !== parseInt(String(req.files.length))) {
	Responses.error(
	res,
	'Mixing of JSONL and non-JSONL files not allowed.',
	StatusCode.INTERNAL_ERROR,
	StatusMessage.UPLOAD_ERROR
	);
	return;
	}
	if (files.filter((file: Express.Multer.File) => path.extname(file.originalname) === '.jsonl').length === 0) {
	// Convert files to JSONL, then concatenate and upload to OpenAI API
	const concatTxt2Jsonlines$ = from(this.proxy.concatTxt2Jsonlines(files, req.body.titles));
	concatTxt2Jsonlines$.pipe(
	mergeMap((concatenatedFile: Express.Multer.File) => {
	const jsonlDataReadStream = fs.createReadStream(concatenatedFile.path, { encoding: 'utf8' });
	return this.proxy.uploadFile(jsonlDataReadStream, 'fine-tune');
	})
	).subscribe({
	next: (response) => {
	console.log('Received `uploadFile()` response');
	if (response.status === StatusCode.SUCCESS) {
	Responses.success(
	res,
	{ response: response.data, responseConfig: response.config.data },
	StatusMessage.RECEIVED_OPENAI_API_RESPONSE
	);
	} else {
	Responses.error(res, 'Upload error', response.status, StatusMessage.GATEWAY_ERROR);
	}
	},
	error: (err) => {
	console.log(err);
	if (err instanceof Error) {
	Responses.error(res, err.message, StatusCode.INTERNAL_ERROR, StatusMessage.BAD_REQUEST);
	}
	},
	complete: () => console.log('Complete')
	});
	} else {
	// Concatenate uploaded files into single JSONL file and upload to OpenAI API
	}
	} catch (err: any) {
	console.log(err);
	if (err instanceof Error) {
	Responses.error(res, err.message, StatusCode.INTERNAL_ERROR, StatusMessage.BAD_REQUEST);
	}
	}
	}
	/**
	* A list of files that belong to the user's organization
	*
	* @param req
	* @param res
	*/
	protected async listFiles(req: express.Request, res: express.Response) {
	try {
	// @ts-ignore
	const response = await this.proxy.openai.listFiles();
	if (response.status === StatusCode.SUCCESS) {
	Responses.success(
	res,
	{ response: response.data, responseConfig: response.config.data },
	StatusMessage.RECEIVED_OPENAI_API_RESPONSE
	);
	}
	} catch (err: any) {
	console.log(err);
	Responses.serverError(res);
	}
	}
	/**
	* Delete file from OpenAI storage
	*
	* @param req
	* @param res
	*
	* TODO: Add input validation
	*/
	protected async deleteFile(req: express.Request, res: express.Response) {
	const fileId = String(req.params.file_id);
	let response;
	try {
	// @ts-ignore
	response = await this.proxy.deleteFile(fileId);
	if (response.status === StatusCode.SUCCESS) {
	Responses.success(
	res,
	{ response: response.data, responseConfig: response.config.data },
	StatusMessage.RECEIVED_OPENAI_API_RESPONSE
	);
	}
	} catch (err: any) {
	if (err.message === 'Request failed with status code 404') {
	Responses.error(res, 'File not found.', StatusCode.NOT_FOUND, StatusMessage.NOT_FOUND);
	}
	// console.log(err);
	Responses.error(res, err.message, StatusCode.INTERNAL_ERROR, StatusMessage.INTERNAL_SERVER_ERROR);
	}
	}
	/**
	* Get information about single previously uploaded JSONL file from the OpenAI storage
	*
	* @param req
	* @param res
	*
	* TODO: Add input validation
	*/
	protected async getFile(req: express.Request, res: express.Response) {
	const fileId = String(req.openaiFileId);

I also found some related docs:

Summary of related docs from https://platform.openai.com/docs/:

To implement uploading of files for model fine-tuning to the OpenAI API, you can follow these steps:

1. In the SoakpProxy.ts file, add a new method called uploadFineTuneFile that takes a file and a purpose as parameters. This method should use the OpenAI API's createFile method to upload the file. The purpose should be set to 'fine-tune'.

2. In the OpenaiFinetunesApi.ts file, add a new endpoint called POST /openai/fine-tunes/upload that handles the HTTP request for file upload. This endpoint should call the uploadFineTuneFile method in the SoakpProxy class to upload the file to the OpenAI API.

3. Convert the uploaded files into JSONL format using Python means.

By following these steps, you will be able to implement the uploading of files for model fine-tuning to the OpenAI API.

Sources:

• https://platform.openai.com/docs/api-reference/files

• https://platform.openai.com/docs/api-reference/embeddings

• https://platform.openai.com/docs/api-reference/chat/object

• https://platform.openai.com/docs/api-reference/completions/create

• https://platform.openai.com/docs/api-reference/models/object

---

Step 2: 🧐 Snippet Analysis

From looking through the relevant snippets, I decided to make the following modifications:

File Path	Proposed Changes
src/SoakpProxy.ts	Modify src/SoakpProxy.ts with contents: • Add a new method uploadFineTuneFile that takes a file and a purpose as parameters. This method should use the OpenAI API's createFile method to upload the file. The purpose should be 'fine-tune'.
src/openai/OpenaiFinetunesApi.ts	Modify src/openai/OpenaiFinetunesApi.ts with contents: • Add a new endpoint POST /openai/fine-tunes/upload that handles the HTTP request for file upload. This endpoint should call the uploadFineTuneFile method in the SoakpProxy class to upload the file to the OpenAI API.
src/openai/OpenaiFilesApi.ts	Modify src/openai/OpenaiFilesApi.ts with contents: • Add a new method convertToJSONL that takes a file as a parameter and converts it into JSONL format using Python means. This method should be called before the file is uploaded to the OpenAI API.

---

Step 3: 📝 Planning

I have created a plan for writing the pull request. I am now working my plan and coding the required changes to address this issue. Here is the planned pull request:

Implement uploading of files for model fine-tuning to OpenAI API
sweep/upload-files-for-fine-tuning

Description

This PR implements the functionality to upload files for model fine-tuning to the OpenAI API. It adds a new method uploadFineTuneFile to the SoakpProxy class, which uses the OpenAI API's createFile method to upload the file. The purpose is set to 'fine-tune'. Additionally, a new endpoint POST /openai/fine-tunes/upload is added to the OpenaiFinetunesApi class, which handles the HTTP request for file upload and calls the uploadFineTuneFile method in the SoakpProxy class.

To ensure compatibility with the OpenAI API, the uploaded files are converted into JSONL format using Python means. The conversion functionality is added to the OpenaiFilesApi class.

Summary of Changes

• Added uploadFineTuneFile method to SoakpProxy class to upload files for model fine-tuning to the OpenAI API.
• Added POST /openai/fine-tunes/upload endpoint to OpenaiFinetunesApi class to handle file upload requests.
• Added convertToJSONL method to OpenaiFilesApi class to convert uploaded files into JSONL format.

---

Step 4: ⌨️ Coding

File	Instructions	Progress
src/SoakpProxy.ts	Modify src/SoakpProxy.ts with contents: • Add a new method uploadFineTuneFile that takes a file and a purpose as parameters. This method should use the OpenAI API's createFile method to upload the file. The purpose should be 'fine-tune'.	✅ Commit 2ff492c
src/openai/OpenaiFinetunesApi.ts	Modify src/openai/OpenaiFinetunesApi.ts with contents: • Add a new endpoint POST /openai/fine-tunes/upload that handles the HTTP request for file upload. This endpoint should call the uploadFineTuneFile method in the SoakpProxy class to upload the file to the OpenAI API.	✅ Commit 2f4bc14
src/openai/OpenaiFilesApi.ts	Modify src/openai/OpenaiFilesApi.ts with contents: • Add a new method convertToJSONL that takes a file as a parameter and converts it into JSONL format using Python means. This method should be called before the file is uploaded to the OpenAI API.	✅ Commit e6d29bf

---

Step 5: 🔁 Code Review

Here are my self-reviews of my changes at sweep/upload-files-for-fine-tuning.

Here is the 1st review

Thanks for your contribution. There are a few changes that need to be made:

• In src/SoakpProxy.ts, please ensure that openai is properly imported or is a property of the SoakpProxy class. Also, consider improving the error handling in the uploadFineTuneFile method to handle errors other than TypeError. You can find these changes on lines 394-410.

• In src/openai/OpenaiFilesApi.ts, the convertToJSONL method is currently unimplemented. Please add the Python code to convert the file to JSONL format as indicated in the comment. This change is required on lines 386-395.

• In src/openai/OpenaiFinetunesApi.ts, please ensure that the uploadFineTuneFile method and the middleware functions validateToken, getKeyStorage, getUser, getProxyInstance are properly implemented and are available in the context ctx. These changes are required on lines 60-65.

Please make these changes and update the pull request. If you need any help, feel free to ask.

I finished incorporating these changes.

---

To recreate the pull request edit the issue title or description.
^{Join Our Discord}