Setup
-----

- Set up conda environment with pip install:
	conda create -n gpt_batch_api python=3.12
	conda activate gpt_batch_api
	pip install gpt_batch_api
	pip check

- Set up conda environment for directly using the library source:
	conda create -n gpt_batch_api python=3.12
	conda activate gpt_batch_api
	conda install -c conda-forge filelock pillow 'pydantic>=2' pytest requests
	pip install openai tiktoken wandb wandb-workspaces
	pip check

- Wandb login:
	wandb login

Run Commands
------------

- General:
	cd /path/to/gpt_batch_api/..  # <-- NOTE: Parent directory of gpt_batch_api (if directly using the library source OR use PYTHONPATH instead)
	conda activate gpt_batch_api
	export OPENAI_API_KEY=sk-...  # <-- NOTE: Replace with actual OpenAI API key
	export WANDB_API_KEY=...  # <-- NOTE: Replace with actual wandb API key (see https://wandb.ai/authorize)

- Wandb logging:
	Wandb is by default used by the TaskManager/GPTRequester classes, unless explicitly disabled using wandb=False (--no_wandb).
	You can control the wandb project name using wand_project='NAME' (--wandb_project NAME, otherwise default is 'gpt_batch_api'), and if need be the wandb entity using wandb_entity='NAME' (--wandb_entity NAME).
	If you wish to resume and continue a previous wandb run (essentially append to it), then find out its run ID (e.g. a1dr4wwa) and use something like wandb_run_id='a1dr4wwa' (--wandb_run_id a1dr4wwa).
	In order to get the most out of the wandb logging, it is recommended to use the preconfigured gpt_batch_api saved view for the project being logged to (to show and organize the most useful stuff to see/monitor):
		Refer to the following URL to see how the saved view will look like:
			https://wandb.ai/pallgeuer/gpt_batch_api_demo?nw=u38mn3ltgfn
		Ensure your target wandb project exists and has at least one run in it => Easiest is just to start your first desired run
		Copy the preconfigured gpt_batch_api saved view to the project workspace (assuming the project is ENTITY/PROJECT):
			python -m gpt_batch_api.wandb_configure_view --dst_entity ENTITY --dst_project PROJECT
		In most cases if the default PROJECT name of 'gpt_batch_api' is being used then you can just do:
			python -m gpt_batch_api.wandb_configure_view --dst_entity ENTITY
		Refer to the help for a more general understanding of what the script can do:
			python -m gpt_batch_api.wandb_configure_view --help
		Open your target wandb project in the web browser:
			https://wandb.ai/ENTITY/PROJECT
		Click on the icon with the three horizontal lines at the top-left of the workspace and switch to the saved view called 'GPT Batch API' (should exist if the wandb_configure_view script worked correctly)
		Click the 'Copy to my workspace' button at the top-right of the workspace => This updates your personal workspace in the project to look like the desired preconfigured view
	You can search the code for the name of logged variables (possibly including the prefix like 'Batch/' for example) to get an idea of what the variable represents (e.g. search for 'num_final_none').

- General configuration (update to suit):
	MODELARGS=(--model gpt-4o-mini-2024-07-18 --cost_input_direct_mtoken 0.150 --cost_input_cached_mtoken 0.075 --cost_input_batch_mtoken 0.075 --cost_output_direct_mtoken 0.600 --cost_output_batch_mtoken 0.300)

- Run the available pytests:
	Command:
		pytest -v gpt_batch_api/utils_test.py
	Verify:
		Verify that all tests passed

- Test the token counting class TokenEstimator:
	Command:
		python -m gpt_batch_api.tokens_test
	Verify:
		Verify that all predicted token totals are equal or close to the actual required number of tokens
		OpenAI changes things from time to time, so TokenEstimator may occasionally require updates in order to be accurate, especially for new models

- Demo the TaskManager class:
	Command:
		python -m gpt_batch_api.task_manager_demo --help
		python -m gpt_batch_api.task_manager_demo --task char_codes "${MODELARGS[@]}"
		python -m gpt_batch_api.task_manager_demo --task utterance_emotion "${MODELARGS[@]}"
	Args:
		Refer to --help
		If you wish to change the directory in which files are created to store the ongoing and final state, and output data of the command (recommended if installed via pip), use:
			--task_dir /path/to/dir
		If you wish not to use wandb for logging and monitoring, use:
			--no_wandb
	Output:
		gpt_batch_api/tasks/char_codes_*
		gpt_batch_api/tasks/utterance_emotion_*
		Associated wandb run in web browser (useful for monitoring progress)
	Verify:
		Verify that the final data output file(s) contain reasonable and complete data (e.g. gpt_batch_api/tasks/char_codes_output*, gpt_batch_api/tasks/utterance_emotion_output*)

- Safe steps when trying out a new custom task:
	Notes:
		It is assumed for these commands that the custom task manager has been wrapped into a script that uses argparse (including also --model) to configure the task manager and GPT requester.
		If hydra or a direct programmatic interface is being used instead for the configuration parameters, then the commands below can be easily adjusted to the appropriate form.
		The generic command is 'python ...', which could for example be 'python -m gpt_batch_api.task_manager_demo --task char_codes'.
	List all available command line arguments:
		python ... --help
	A) [Assuming task does not exist yet...] Initialize a new task but don't run it (task metadata is taken and fixed beyond here unless using --reinit_meta):
		python ... --no_wandb "${MODELARGS[@]}" --no_run [--max_completion_tokens NUM] [--completion_ratio RATIO] [--temperature TEMP] [--top_p MASS] [--opinions_min NUM] [--opinions_max NUM] [--confidence RATIO] ...
	B) Generate a batch of 100 requests without actually executing the batch:
		python ... --no_wandb "${MODELARGS[@]}" --max_session_requests 0 --max_batch_requests 100 --max_unpushed_batches 1
	C) Show 10 verbose examples of generated requests without actually executing them (requires that some batches have already been generated and saved to disk):
		python ... --no_wandb "${MODELARGS[@]}" --force_direct --direct_verbose always --max_session_requests 10 --max_batch_requests 100 --max_unpushed_batches 1 --dryrun
	D) Show 10 verbose examples of generated requests and responses using the direct API:
		python ... --no_wandb "${MODELARGS[@]}" --force_direct --direct_verbose always --max_session_requests 10 --max_batch_requests 100 --max_unpushed_batches 1
	E) Execute 100 requests using the direct API, and show those examples verbosely that had a warning or error:
		python ... --no_wandb "${MODELARGS[@]}" --force_direct --direct_verbose warn --max_session_requests 100 --max_batch_requests 100 --max_unpushed_batches 1
	F) Execute 250 requests using the batch API:
		python ... --no_wandb "${MODELARGS[@]}" --max_batch_requests 250 --max_session_requests 400 --max_unpushed_batches 1
		# <-- Search (where # is some number) the command outputs for 'max # completion tokens' (overall metrics) and 'max # tokens' (per batch), and decide on suitable values for request max_completion_tokens
		# <-- Search the command outputs for 'completion ratio' (overall metrics and per batch), and decide on a suitable value for completion_ratio (keeping in mind/factoring in whatever change you just made to max_completion_tokens of course!)
		python ... --no_wandb "${MODELARGS[@]}" --no_run --reinit_meta --max_completion_tokens NUM --completion_ratio RATIO [--temperature TEMP] [--top_p MASS] [--opinions_min NUM] [--opinions_max NUM] [--confidence RATIO] ...  # <-- CAUTION: Other than updating max_completion_tokens and completion_ratio, make sure to be consistent with A) as --reinit_meta always updates ALL task meta variables
	G) Execute 1 USD worth of fresh (thus wipe ongoing) requests using the batch API (or 1 of whatever currency/unit MODELARGS uses):
		python ... --no_wandb "${MODELARGS[@]}" --only_process  # <-- If there are any unfinished batches busy on the remote (we don't want to unnecessarily wipe them)
		python ... --no_wandb --wipe_requests "${MODELARGS[@]}" --max_batch_cost 1.00 --max_session_cost 1.00 --max_unpushed_batches 1
		# <-- Refine completion ratio value if required (search the command outputs for the 'completion ratio' of any newly executed batches, and reinit meta as in F)
	H) Reset the entire task to scratch and throw away any results obtained so far:
		python ... --no_wandb --wipe_task "${MODELARGS[@]}" --no_run
	Run the task normally to completion (preferably enable wandb by omitting any --no_wandb, change MY_BATCH_TASK to a real name), respecting a batch queue limit of 60,000,000 tokens (= 60000 ktokens) at a time (refer to the OpenAI tier rate limits):
		python ... --wandb_name MY_BATCH_TASK "${MODELARGS[@]}" --max_remote_ktokens 60000
		# <-- The above command can be interrupted with Ctrl+C and restarted at any time, and it will safely and robustly just pick up where it left off without losing anything (assuming the custom task has been correctly and revertibly implemented like described in the documentation, and demonstrated in task_manager_demo.py)
	Give failed samples another chance with up to 4 more retries = 5 more attempts allowed, resuming the previous wandb run (given by RUN_ID) in terms of logging:
		python ... --wandb_name MY_BATCH_TASK --wandb_run_id RUN_ID --wipe_failed "${MODELARGS[@]}" --max_remote_ktokens 60000 --max_retries 4
		# <-- You could also try the command above with for example an extra '--reinit_meta --temperature 0.9 ...' to change the temperature => CAUTION: Just make sure though to remain consistent with A) and F) in all other meta variables, as --reinit_meta always updates ALL task meta variables
	Get the current status of a task without actually executing task steps:
		python ... "${MODELARGS[@]}" --max_remote_ktokens 60000 --no_run

- Commonly useful arguments for debugging, applying code changes, recovering from errors/batch failures, and such:
	--task_dir /PATH/TO/DIR
		Change the directory in which files are created to store the ongoing and final state, and output data of the command (recommended if installed via pip)
	--no_wandb
		Disable wandb logging
	--wandb_project NAME
		Set the desired target wandb project for logging
	--wandb_name NAME
		Set a custom fixed name for the target wandb logging run
	--wandb_run_id RUN_ID
		Resume/append to a wandb run (specified by the desired run ID), or create it if it does not exist (auto-generated run IDs are typically lowercase alphanumeric strings of length 8)
	--dryrun
		Prevent any API calls or changes to saved disk state, and just show what would be done based on the current task state
	--force_direct --direct_verbose always --max_session_requests 20
		Force only the direct API to be used for 20 requests, and print the requests and responses verbosely for debugging purposes (when combined with --dryrun only prints the requests that would have been made)
	--max_remote_ktokens 90 --max_batch_ktokens 90
		Configure how many kilo-tokens can be pending in the batch queue at any one time in order to respect the OpenAI usage tier limitations (Note: max_batch_ktokens must always be less than or equal to max_remote_ktokens, otherwise no batch would fit on the remote)
	--max_batch_requests 50
		Limit the batch size in terms of number of requests per batch (also, num batches/num requests/num tokens/cost/MB size can be limited for each batch, the remote server, each session, or the entire task, as appropriate)
	--max_remote_batches 0 --max_unpushed_batches 3
		Generate up to 3 local batches without letting any of them be pushed (e.g. allowing them to be manually inspected on disk without pushing)
	--max_retries 5
		Adjust how often a request is automatically retried (if a retryable error occurs) before it is declared as failed
	--min_pass_ratio 0.8
		At minimum what ratio of the requests in a batch need to be successful in order for the batch to be declared as 'passed' (too many consecutive non-passed batches for safety lead to an error and aborting the run, see --max_pass_failures 2)
	--process_failed_batches 2 [--retry_fatal_requests]
		If there are failed batches then the task aborts for safety (as manual intervention/code changes/sanity checking is probably required) => This parameter allows up to a certain number of failed batches to be force-processed anyway, thereby allowing the task to proceed and potentially recover. If --retry_fatal_requests is also supplied, then requests that received fatal errors will be allowed to be retried (normally they are not, as fatal errors are ones where it is not expected that a retry has a chance of resolving the issue).
	--only_process
		Wait for and allow all pushed batches to complete and be processed without generating, commiting or pushing any new requests (i.e. no new work is generated or scheduled)
	--reinit_meta
		Force the task metadata to be updated (usually task metadata is only captured once when the task is created/initialized), for example to change the request model, temperature, hyperparameters for parsing, or such (you must ensure that your task implementation can deal with whatever of these parameters you are changing)
	--wipe_requests
		Wipe all ongoing requests (e.g. useful if you changed the request generation and want to reconstruct all the requests in the queue, local batches, and remote) => It is recommended to execute a separate run first with --only_process to avoid unnecessarily losing already-pushed requests and responses
	--wipe_failed
		Wipe all ongoing requests and failed samples, allowing them to be attempted again (with the full number of retries available again) => It is recommended to execute a separate run first with --only_process to avoid unnecessarily losing already-pushed requests and responses
	--wipe_task
		Wipe entire task and start completely from scratch (can be combined with --no_run in order to not actually run the task after wiping)
