Spark Input Formats And Parallel factor

Spark Input Formats And Parallel factor

In Spark RDD/DataFrames have a parallel factor which they inherit from their parent RDD/DataFrame.In case of ReduceByKey, combineByKey it can be set explicitly too using coalesce() or repartition().


The Question arises how does spark decide the parallel factor of the first RDD ?


Spark Operation                 File Input Format  in Spark  Code
      Text File      :       javaSparkContext.textFile(inputPath);     hadoopFile(path, classOf[TextInputFormat],
classOf[LongWritable], classOf[Text], minPartitions)

Sequence File :


  val inputFormatClass = classOf[SequenceFileInputFormat[K, V]]

   hadoopFile(path, inputFormatClass, keyClass, valueClass, minPartitions)



   new WholeTextFileRDD(this, classOf[WholeTextFileInputFormat],   classOf[Text], classOf[Text], updateConf, minPartitions)
ParquetFile :     ParquetFileFormat


The parallel factor of the first RDD/DataFrame is decide by the input split which is decided by input format used in the spark code . Each input function call (textFile, sequenceFile, parquet ) maps to appropriate InputFormat in the spark scala code

How to use a custom inputFormat in spark :

JavaRDD profileJavaRDD = sparkContext.newAPIHadoopFile(inputPath, CustomtInputFormat.class, KEY.class, VALUE.class, sparkContext.hadoopConfiguration())