Spring Batch Job Restart: What should be the correct values for chunk and readerPageSize with respect to grid size.

[PSHREYASHOLLA]

This is posted here as I did not get a response on https://stackoverflow.com/questions/78199004/restarting-a-failed-job-is-not-processing-the-failed-chunk-data-again-continua

This is my step definition for spring batch partition approach,

@bean public Step step1() { return new StepBuilder("step1", jobRepository) .partitioner(slaveStep().getName(), partitioner()) .step(slaveStep()) .gridSize(febpEmployeeTaxCalculationBatchProperties.getTaxCalculationStep1PartitionGridSize()) .taskExecutor(actStmntTaskExecutor()) .build(); }

// slave step
@Bean
public Step slaveStep() 
{
    
    try {
        return new StepBuilder("slaveStep", jobRepository)
                .<EmployeeDetail, EmployeeTaxDetail>chunk(febpEmployeeTaxCalculationBatchProperties.getTaxCalculationStep1ReaderPageSize(),transactionManager)
                .reader(pagingItemReader(null,null))
                .processor(processor())
                .writer(customerItemWriter())
            .taskExecutor(actStmntTaskExecutor())
                .build();
    } catch (Exception ex) {
        throw new RuntimeException("Error creating slave step: " + ex.getMessage());
    }
}

/**

• Act stmnt task executor.
• @return the simple async task executor
  */
  public SimpleAsyncTaskExecutor actStmntTaskExecutor() {
  SimpleAsyncTaskExecutor acctStmtTaskExecuter = new SimpleAsyncTaskExecutor();
  acctStmtTaskExecuter.setConcurrencyLimit(febpEmployeeTaxCalculationBatchProperties.getTaxCalculationStep1TaskExecuterThreadConcurrencyLimit());
  acctStmtTaskExecuter.setThreadPriority(febpEmployeeTaxCalculationBatchProperties.getTaxCalculationStep1TaskExecuterThreadPriority());
  acctStmtTaskExecuter.setThreadNamePrefix("FEBP_TAX_CALCULATION_GEN");
  return acctStmtTaskExecuter;
  }

So here for restart to work,

1)Recommendation for restart is not using TaskExecuter. So should we not use TaskExecuter as mentioned above in step1() and slaveStep()? How to achieve concurrency ?

2. So only SimpleAsyncTaskExecutor needs to be used? If yes is it at step or slavestep?

3)How should the chunk size, gridSize and readers pagesize and fetchsiz be calculated? As I see if chunk size is set to low values, restart is not re-writting failed data.

[PSHREYASHOLLA]

Team any update for this?

[fmbenhassine]

Gird size is a hint to the partitioner about the number of partitions you want to create.

The chunk size, as you might be familiar with, is the number of items that will be processed as a unit in a single transaction.

The page size is the number of items that the reader should read in a single page.

Those are different parameters and the is no relation between them.

It is up to you to find the best values for each. For example, if your dataset is composed of 10.000 records and the grid size = 10, then you will have 10 partitions with 1000 each. Now since each worker will be assigned 1000 records, you could set the page size = chunk size = 100 for instance (to have a single page read per chunk).

I am turning this into a discussion, and let's continue there.