python multiprocessing - optimize loop making django-rest-framework requests -

i have loop iterates on array. each item in array, calls function makes django-rest-framework requests. each function call independent of others.

if array has 25 items, takes 30 seconds complete. trying total time down less 10 seconds.

half time spent in function taken drf requests. make sense replace loop multiprocessing pool? if so, how ensure each process makes requests on separate connection using requests package?

i tried replacing:

for scenario_id in scenario_ids:     step_scenario_partial(scenario_id) 

with:

pool = pool(processes=2) pool.map(step_scenario_partial, scenario_ids) 

which failed due openssl.ssl.error: [('ssl routines', 'ssl3_get_record', 'decryption failed or bad record mac')]

according this, error due re-using same ssl connection in more 1 process.

you can use concurrent python module (docs) can execute parallel tasks. example method returns list of response objects:

from concurrent import futures  def execute_all(scenario_ids, num_workers=5):     '''     method make parallel api calls     '''     futures.threadpoolexecutor(max_workers=num_workers) executor:         return [result result in executor.map(step_scenario_partial, scenario_ids)] 

the threadpoolexecutor uses pool of threads execute asynchronous parallel calls. can experiment values of num_workers, starting 5, ensure total execution time <10 seconds.