API¶
The great flexibility of imc
comes from the ability to compose workflows using the API.
It provides a rich but abstract imc.analysis.Project
object and implements various modules building on it depending on the data type.
In addition, the imc.operations
module contains several analysis-independent methods and the imc.utils
module provides low-level functions of general use.
imc.data_models.project¶
A class to model a imaging mass cytometry project.
- class imc.data_models.project.Project(metadata=None, name='project', sample_name_attribute='sample_name', sample_grouping_attributes=None, panel_metadata=None, toggle=True, subfolder_per_sample=True, processed_dir=Path('processed'), results_dir=Path('results'), **kwargs)[source]¶
A class to model a IMC project.
- channel_correlation(channel_exclude=None, samples=None, rois=None)[source]¶
Observe the pairwise correlation of channels across ROIs.
- Return type
- channel_summary(red_func='mean', channel_exclude=None, plot=True, output_prefix=None, samples=None, rois=None, **kwargs)[source]¶
Summary statistics on the signal of each channel across ROIs.
- cluster_cells(output_prefix=None, plot=True, set_attribute=True, samples=None, rois=None, **kwargs)[source]¶
Derive clusters of single cells based on their channel intensity.
- find_communities(output_prefix=None, samples=None, rois=None, **kwargs)[source]¶
Find communities and supercommunities of cell types across all images.
- Return type
- get_input_filename(input_type)[source]¶
Get path to file with data for Sample.
- Available input_type values are:
“cell_type_assignments”: CSV file with cell type assignemts for each cell and each ROI
- Return type
- label_clusters(h5ad_file=None, output_prefix=None, **kwargs)[source]¶
Derive labels for each identified cluster based on its most abundant markers.
- Return type
- measure_adjacency(output_prefix=None, samples=None, rois=None)[source]¶
Derive cell adjacency graphs for each ROI.
- Return type
- plot_channels(channels=['mean'], per_sample=False, merged=False, save=False, output_dir=None, samples=None, rois=None, **kwargs)[source]¶
Plot a tp.List of channels for all Samples/ROIs.
- Return type
- quantify_cell_intensity(samples=None, rois=None, **kwargs)[source]¶
Measure the intensity of each channel in each single cell.
- Parameters
samples (Sequence[Sample]) – Subset of samples to use. Default is all.
rois (Sequence[ROI]) – Subset of samples to use. Default is all.
kwargs – Additional keyword-arguments will be passed to imc.ops.quant.quantify_cell_intensity: red_func: str
Function to reduce values per cell. Default is “mean”.
- border_objs: bool
Whether to quantify objects touching image border. Default is False.
- equalize: bool
Whether to scale the signal. This is actually a cap on the 98th percentile. Default is True. TODO: change keyword name to ‘percentile_scale’.
- scale: bool
Whether to scale signal to unit space. Default is False.
- channel_include: Array
Sequence of channels to include. This is a boolean array matching the ROI channels. Default is None: all channels.
- channel_exclude: Array
Sequence of channels to exclude. This is a boolean array matching the ROI channels. Default is None: no channels.
- Return type
- quantify_cell_morphology(samples=None, rois=None, **kwargs)[source]¶
Measure the shape parameters of each single cell.
- Parameters
samples (Sequence[Sample]) – Subset of samples to use. Default is all.
rois (Sequence[ROI]) – Subset of samples to use. Default is all.
kwargs – Additional keyword-arguments will be passed to imc.ops.quant.quantify_cell_morphology:
- attributes: Sequence[str]
Attributes to quantify. For extensive list refer to https://scikit-image.org/docs/dev/api/skimage.measure.html#skimage.measure.regionprops
- border_objs: bool
Whether to quantify objects touching image border. Default is False.
- Return type
- quantify_cells(layers: List[str], intensity: bool, intensity_kwargs: Dict[str, Any], morphology: bool, morphology_kwargs: Dict[str, Any], set_attribute: Literal[True], samples: List[imc.data_models.sample.IMCSample], rois: List[imc.data_models.roi.ROI]) None [source]¶
- quantify_cells(layers: List[str], intensity: bool, intensity_kwargs: Dict[str, Any], morphology: bool, morphology_kwargs: Dict[str, Any], set_attribute: Literal[False], samples: List[imc.data_models.sample.IMCSample], rois: List[imc.data_models.roi.ROI]) pandas.core.frame.DataFrame
Measure the channel and morphological features of each single cell.
- property rois: List[imc.data_models.roi.ROI]¶
Return a tp.List of all ROIs of the project samples.
imc.data_models.sample¶
A class to model a imaging mass cytometry sample.
- class imc.data_models.sample.IMCSample(sample_name='sample', root_dir=None, metadata=None, subfolder_per_sample=True, roi_name_atribute='roi_name', roi_number_atribute='roi_number', panel_metadata=None, channel_labels=None, prj=None, **kwargs)[source]¶
If metadata is given, it will initialize ROI objects for each row.
If panel_metadata is given, it will use that
- cluster_cells(output_prefix=None, plot=True, set_attribute=True, rois=None, **kwargs)[source]¶
Derive clusters of single cells based on their channel intensity.
- get_input_filename(input_type)[source]¶
Get path to file with data for Sample.
- Available input_type values are:
“cell_type_assignments”: CSV file with cell type assignemts for each cell and each ROI
- Return type
- plot_channels(channels=['mean'], merged=False, rois=None, per_roi=False, save=False, output_dir=None, **kwargs)[source]¶
Plot a list of channels for all ROIs.
- Return type
- quantify_cell_intensity(rois=None, **kwargs)[source]¶
Measure the intensity of each channel in each single cell.
- Return type
- quantify_cell_morphology(rois=None, **kwargs)[source]¶
Measure the shape parameters of each single cell.
- Return type
imc.data_models.roi¶
A class to model a imaging mass cytometry acquired region of interest (ROI).
- class imc.data_models.roi.ROI(name='roi', roi_number=None, channel_labels=None, root_dir=None, stacks_dir=Path('tiffs'), masks_dir=Path('tiffs'), single_cell_dir=Path('single_cell'), sample=None, default_mask_layer='cell', **kwargs)[source]¶
A class to model a region of interest in an IMC experiment.
- property cell_mask: numpy.ndarray¶
An array with unique integers for each cell, excluding cells touching image borders.
- Return type
- property cell_mask_o: numpy.ndarray¶
An array with unique integers for each cell. Original array including cells touching image borders.
- Return type
- property channel_labels: pandas.core.series.Series¶
Return a Series with a string for each channel in the ROIs stack.
- Return type
- property cytoplasm_mask: numpy.ndarray¶
An array with unique integers for the cytoplasm of each cell. The cytoplasm is defined as the cell area excluding nuclei and membrane.
- Return type
- property extracellular_mask: numpy.ndarray¶
An array with unique integers for the extracellular area of each cell. The extracellular area is a fixed amount around the cell, not overlapping other cells.
- Return type
- get_distinct_marker_sets(n_groups=4, group_size=4, save_plot=False)[source]¶
Use cross-channel correlation to pick n clusters of distinct channels to overlay
- get_input_filename(input_type)[source]¶
Get path to file with data for ROI.
- Available input_type values are:
“stack”: Multiplexed image stack
“channel_labels”: Labels of channels (may not exist for OME-TIFF stacks)
“ilastik_input”: Features extracted by ilastik (usually not available by default)
“probabilities”: 3 color probability intensities predicted by ilastik
“cell_mask”: TIFF file with mask for cells
“nuclei_mask”: TIFF file with mask for nuclei
“nuclear_mask”: TIFF file with mask for nuclei
“cell_type_assignments”: CSV file with cell type assignemts for each cell
“adjacency_graph”: Cell neighborhood graph.
- Return type
- property mask: numpy.ndarray¶
An array with unique integers for the default mask area of each cell.
- Return type
- property membrane_mask: numpy.ndarray¶
An array with unique integers for the membrane of each cell. The membrane is the area of cell defined by a fixed border in the cell’s border.
- Return type
- property nuclei_mask: numpy.ndarray¶
An array with unique integers for each nuclei matched with cell mask not touching image borders.
- Return type
- property nuclei_mask_o: numpy.ndarray¶
An array with unique integers for each cell. Original array including nuclei of cells touching image borders.
- Return type
- plot_cell_types(cell_type_assignments=None, cell_type_combinations=None, position=None, ax=None, palette=None, add_scale=True, add_legend=True, legend_kwargs={})[source]¶
If ax is given it must match number of cell_type_combinations.
- plot_channel(channel, ax=None, equalize=True, log=True, minmax=True, smooth=None, position=None, add_scale=True, add_range=True, **kwargs)[source]¶
Plot a single channel.
Supports indexing of channels either by name or integer. Special strings for
numpy.ndarray
functions can be passed to reduce values across channels (first axis). Pass e.g. ‘mean’ or ‘sum’.Keyword arguments are passed to
imshow()
- Return type
- plot_channels(channels=None, merged=False, axes=None, equalize=None, log=True, minmax=True, smooth=None, position=None, add_scale=True, add_range=True, share_axes=True, **kwargs)[source]¶
If axes is given it must be length channels
- **kwargs: dict
Additional keyword arguments will be passed to imc.graphics.merge_channels. Pass ‘target_colors’ to select colors to use when using merged.
- plot_overlayied_channels_subplots(n_groups)[source]¶
Plot all channels of ROI in n_groups combinations, where each combination has as little overlap as possible.
- Return type
- plot_probabilities_and_segmentation(axes=None, add_scale=True)[source]¶
Visualize channel mean, DNA channel, segmentation probabilities and the segmented nuclei and cells.
If axes is given it must have length 5
- quantify_cell_intensity(channel_include=None, channel_exclude=None, layers=['cell'], **kwargs)[source]¶
Quantify intensity of each cell in each channel.
- Return type
- quantify_cell_morphology(layers=['cell'], **kwargs)[source]¶
Quantify shape attributes of each cell. Additional keyword arguments are passed to imc.ops.quant.quantify_cell_morphology.
- Return type
- read_all_inputs(only_these_keys=None, permissive=False, set_attribute=True, overwrite=False, parameters=None)[source]¶
- Reads in all sample-wise inputs:
raw stack
extracted features
probabilities
uncertainty
segmentation mask.
If permissive is
True
, skips non-existing inputs.
- read_input(key, permissive=False, set_attribute=True, overwrite=False, parameters=None)[source]¶
- Reads in all sample-wise inputs:
raw stack
extracted features
probabilities
uncertainty
segmentation mask.
If permissive is
True
, skips non-existing inputs.
- property stack: numpy.ndarray¶
An ndarray representing the image channel stack.
- Return type
- property stack_eq¶
Same as stack but equalized per channel.
imc.operations¶
imc.ops.signal¶
Functions for handling signal intensity in images.
imc.ops.compensation¶
Functions for compensation of imaging mass cytometry data.
imc.ops.mixture¶
Functions for mixtures of signal.
imc.ops.domain¶
Functions for image annotations.
- imc.ops.domain.export_images_for_topological_labeling(rois, output_dir, channels=['mean'], overwrite=False)[source]¶
Export PNGs for labeling with labelme.
- Return type
- imc.ops.domain.get_domain_areas(topo_annots: Dict[str, Dict], rois: Sequence[imc.data_models.roi.ROI], per_domain: Literal[False]) Dict[imc.types.Path, float] [source]¶
- imc.ops.domain.get_domain_areas(topo_annots: Dict[str, Dict], rois: Sequence[imc.data_models.roi.ROI], per_domain: Literal[True]) pandas.core.frame.DataFrame
Get area of airways per image in microns.
- imc.ops.domain.get_domains_per_cell(topo_annots, rois, exclude_domains=None, remaining_domain='background', resolution='largest')[source]¶
Generate annotation of topological domain each cell is contained in based on manual annotated masks.
- Parameters
topo_annots (dict) – Dictionary of annotations for each ROI.
rois (list) – List of ROI objects.
exclude_domains (list[str]) – Domains to ignore
remaining_domain (str | dict[str, str]) – Name of domain to fill in for cells that do not fall under any domain annotation. If given a string, it will simply use that. If given a dict, the filled domain will be the value of the key which exists in the image. E.g. Annotating tumor/stroma domains. If an image has only domains of type ‘Tumor’, given remaining_domain == {‘Tumor’: ‘Stroma’, ‘Stroma’: ‘Tumor’}, the remaining cells will be annotated with ‘Stroma’. In an image annotated only with ‘Stroma’ domains, remaining cells will be annotated with ‘Tumor’ domains.
resolution (str) – If remaining_domain is a dict, there may be more than one domain present in the image. A resolution method is thus needed to select which domain will be filled for the remaining cells. The method ‘largest’ will choose as key of remaining_domain the largest annotated domain class. The method ‘unique’ will be strict and only fill in if there is a unique domain.
- Return type
imc.ops.quant¶
Operations of signal quantification.
- imc.ops.quant.quantify_cell_intensity(stack, mask, red_func='mean', border_objs=False, equalize=True, scale=False, channel_include=None, channel_exclude=None)[source]¶
Measure the intensity of each channel in each cell
- Parameters
stack (tp.Union[Array, Path]) – Image to quantify.
mask (tp.Union[Array, Path]) – Mask to quantify.
red_func (str) – Function to reduce pixels to object borders. Defaults to ‘mean’.
border_objs (bool) – Whether to quantify objects touching image border. Defaults to False.
channel_include (
ndarray
) – Boolean array for channels to include.channel_exclude (
ndarray
) – Boolean array for channels to exclude.
- Return type
- imc.ops.quant.quantify_cell_intensity_rois(rois, **kwargs)[source]¶
Measure the intensity of each channel in each single cell.
- Return type
imc.ops.clustering¶
Functions for single-cell clustering.
imc.ops.adjacency¶
Functions for single-cell adjacency.
- imc.ops.adjacency.get_adjacency_graph(roi, output_prefix=None, max_dist=4)[source]¶
Derive a spatial representation of cells in image using a graph.
- Parameters
roi (imc.ROI) – ROI object to derive graph for.
output_prefix (typing.Path) – Prefix to output file with graph. Defaults to sample root dir / ‘single_cell’.
max_dist (int) – Maximum distance to consider physical interaction between cells (graph edges)
- Returns
Adjacency graph for cells in ROI.
- Return type
networkx.Graph
- imc.ops.adjacency.measure_cell_type_adjacency(roi, method='random', adjacency_graph=None, n_iterations=100, inf_replace_method='min', output_prefix=None, plot=True, save=True)[source]¶
Derive an aggregated measure of adjacency betwen cell types for one ROI.
- Parameters
roi (imc.ROI) – ROI object to derive graph for.
method (str) – Method to normalize interactions by. - ‘random’: generate empirical background of expected interactions based on cell type abundance by randomization (permutation of cell type identities). - ‘pharmacoscopy’: method with analytical solution from Vladimer et al (10.1038/nchembio.2360). Not recommended for small images. Default is ‘random’.
adjacency_graph (networkx.Graph) – Adjacency graph per cell for ROI. By default, and if not given will be the ROI.adjacency_graph attribute.
n_iterations (int) – Number of permutations to run when method == ‘random’. Defaults to 100.
inf_replace_method (str) – If method == ‘pharmacoscopy’, how to handle cases where interactions are not observed.
output_prefix (typing.Path) – Prefix to output file with graph. Defaults to sample root dir / ‘single_cell’.
plot (bool) – Whether to plot visualizations. Default is True.
save (bool) – Whether to save output to disk. Default is True.
- Returns
DataFrame of cell type interactions normalized by method.
- Return type
imc.ops.community¶
Functions for community detection.
imc.graphics¶
Plotting functions and utilities to handle images.
- class imc.graphics.InteractiveViewer(obj, show=False, up_key='w', down_key='s', log_key='l', **kwargs)[source]¶
An interactive image viewer for multiplexed images.
- Parameters
obj (ROI | Array) – An ROI object or a numpy array
**kwargs (dict) – Additional keyword arguments to pass to matplotlib.pyplot.imshow.
- get_slice()[source]¶
Get a array slice for the current index with current transformations.
- Return type
- imc.graphics.add_centroids(a, ax=None, res=None, column=None, algo='umap')[source]¶
a: AnnData ax: matplotlib.Axes.axes res: resolution of clusters to label column: Column to be used. Has precedence over res.
- imc.graphics.add_legend(patches, ax=None, **kwargs)[source]¶
Add a legend to an existing axis.
- Return type
- imc.graphics.add_minmax(minmax, _ax=None)[source]¶
Add an annotation of the min and max values of the array.
- Return type
- imc.graphics.add_scale(_ax=None, width=100, unit='$\\\\mu$m')[source]¶
Add a scale bar to a figure. Should be called after plotting (usually with matplotlib.pyplot.imshow).
- Return type
- imc.graphics.cell_labels_to_mask(mask, labels)[source]¶
Replaces integers in mask with values from the mapping in labels.
- Return type
- imc.graphics.get_grid_dims(dims: Union[int, Collection], return_fig: Literal[True], nstart: Optional[int]) matplotlib.figure.Figure [source]¶
- imc.graphics.get_grid_dims(dims: Union[int, Collection], return_fig: Literal[False], nstart: Optional[int]) Tuple[int, int]
Given a number of dims subplots, choose optimal x/y dimentions of plotting grid maximizing in order to be as square as posible and if not with more columns than rows.
- imc.graphics.get_n_colors(n, max_value=1.0)[source]¶
With modifications from https://stackoverflow.com/a/13781114/1469535
- Return type
- imc.graphics.merge_channels(arr, target_colors=None, return_colors=False)[source]¶
Assumes [0, 1] float array. to is a tuple of 3 colors.
- imc.graphics.plot_single_channel(arr, axis=None, cmap=None)[source]¶
Plot a single image channel either in a new figure or in an existing axis
- imc.graphics.rainbow_text(x, y, strings, colors, orientation='horizontal', ax=None, **kwargs)[source]¶
Take a list of strings and colors and place them next to each other, with text strings[i] being shown in colors[i].
- Parameters
x, y (float) – Text position in data coordinates.
strings (list of str) – The strings to draw.
colors (list of color) – The colors to use.
orientation ({‘horizontal’, ‘vertical’})
ax (Axes, tp.optional) – The Axes to draw into. If None, the current axes will be used.
**kwargs – All other keyword arguments are passed to plt.text(), so you can set the font size, family, etc.
From (https://matplotlib.org/3.2.1/gallery/text_labels_and_annotations/rainbow_text.html)
- imc.graphics.rasterize_scanpy(fig)[source]¶
Rasterize figure containing Scatter plots of single cells such as PCA and UMAP plots drawn by Scanpy.
- Return type
- imc.graphics.saturize(arr)[source]¶
Saturize an image by channel, by minmax scalling each.
- Return type
Share given axes after figure creation. Useful when not all subplots of a figure should be shared.
- Return type
imc.utils¶
Convenience utilities for the package.
- imc.utils.cleanup_channel_names(series)[source]¶
Standardize channel naming using a set of defined rules.
- Return type
- imc.utils.downcast_int(arr, kind='u')[source]¶
Downcast numpy array of integers dependent on largest number in array compatible with smaller bit depth.
- Return type
- imc.utils.download_file(url, output_file, chunk_size=1024)[source]¶
Download a file and write to disk in chunks (not in memory).
- imc.utils.filter_kwargs_by_callable(kwargs, callabl, exclude=None)[source]¶
Filter a dictionary keeping only the keys which are part of a function signature.
- imc.utils.fractal_dimension(Z, threshold=0.9)[source]¶
https://gist.github.com/viveksck/1110dfca01e4ec2c608515f0d5a5b1d1
- imc.utils.get_canny_edge_image(image, mask, radius=30, sigma=0.5)[source]¶
Compute Canny edge image.
- imc.utils.lacunarity(image, box_size=30)[source]¶
From here: https://satsense.readthedocs.io/en/latest/_modules/satsense/features/lacunarity.html Calculate the lacunarity value over an image.
The calculation is performed following these papers:
Kit, Oleksandr, and Matthias Luedeke. “Automated detection of slum area change in Hyderabad, India using multitemporal satellite imagery.” ISPRS journal of photogrammetry and remote sensing 83 (2013): 130-137.
Kit, Oleksandr, Matthias Luedeke, and Diana Reckien. “Texture-based identification of urban slums in Hyderabad, India using remote sensing data.” Applied Geography 32.2 (2012): 660-667.
- imc.utils.minmax_scale(x, by_channel=True)[source]¶
Scale array to 0-1 range.
- x: np.ndarray
Array to scale
- by_channel: bool
Whether to perform scaling by the smallest dimension (channel). Defaults to True.
- imc.utils.plot_panoramas_rois(yaml_spec, output_prefix, panorama_image_prefix=None, save_roi_arrays=False, overwrite=False)[source]¶
Plot the location of panoramas and ROIs of a IMC sample.
- yaml_spec: tp.Union[str, pathlib.Path]
Path to YAML file containing the spec of the acquired sample.
- output_prefix: tp.Union[str, pathlib.Path]
Prefix path to output the joint image and arrays if save_roi_arrays is True.
- panorama_image_prefix: tp.Union[str, pathlib.Path]
Prefix of images of panoramas captured by the Hyperion instrument.
- save_roi_arrays: bool
Whether to output arrays containing the images captured by the Hyperion instrument in the locations of the ROIs.
- Return type
- imc.utils.polygon_to_mask(polygon_vertices, shape, including_edges=True)[source]¶
Convert a set of vertices to a binary array.
Adapted and extended from: https://stackoverflow.com/a/36759414/1469535.
- Return type
- imc.utils.read_image_from_file(file, equalize=False)[source]¶
Read images from a tiff or hdf5 file into a numpy array. Channels, if existing will be in first array dimension. If equalize is
True
, convert to float type bounded at [0, 1].- Return type
- imc.utils.run_shell_command(cmd, dry_run=False, quiet=False)[source]¶
Run a system command.
Will detect whether a separate shell is required.
- Return type
- imc.utils.sorted_nicely(iterable)[source]¶
Sort an iterable in the way that humans expect.
- Parameters
l (iterable) – tp.Sequence to be sorted
- Returns
Sorted iterable
- Return type
iterable
- imc.utils.stack_to_probabilities(stack, channel_labels, nuclear_channels=None, cytoplasm_channels=None, log=True)[source]¶
Very simple way to go from a channel stack to nuclei, cytoplasm and background probabilities.
- Return type
- imc.utils.txt_to_tiff(txt_file, tiff_file, write_channel_labels=True)[source]¶
Convert a Fluidigm TXT file to a TIFF file.
- Parameters
txt_file – Input text file from Fluidigm.
tiff_file – Path to output file.
write_channel_labels – Whether to write a file with labels for the channel names.
- Return type
- imc.utils.write_ometiff(arr, labels, output_path, compression_level=3, description=None, **tiff_kwargs)[source]¶
Write DataArray to a multi-page OME-TIFF file.
- Parameters
arr (np.ndarray) – Array of dimensions CYX.
output_path (str | pathlib.Path) – File to write TIFF file to.
**kwargs – Additional arguments to tifffile.imwrite.
- Return type
- imc.utils.z_score(x: numpy.ndarray, axis: Union[Literal[0], Literal[1]]) numpy.ndarray [source]¶
- imc.utils.z_score(x: pandas.core.frame.DataFrame, axis: Union[Literal[0], Literal[1]]) pandas.core.frame.DataFrame
Standardize and center an array or dataframe.
imc.types¶
Specific types or type aliases used in the library.
- class imc.types.AnnData(X=None, obs=None, var=None, uns=None, obsm=None, varm=None, layers=None, raw=None, dtype='float32', shape=None, filename=None, filemode=None, asview=False, *, obsp=None, varp=None, oidx=None, vidx=None)[source]¶
An annotated data matrix.
AnnData
stores a data matrixX
together with annotations of observationsobs
(obsm
,obsp
), variablesvar
(varm
,varp
), and unstructured annotationsuns
.An
AnnData
object adata can be sliced like aDataFrame
, for instance adata_subset = adata[:, list_of_variable_names].AnnData
’s basic structure is similar to R’s ExpressionSet [Huber15]. If setting an .h5ad-formatted HDF5 backing file .filename, data remains on the disk but is automatically loaded into memory if needed. See this `blog post`_ for more details.- Parameters
X – A #observations × #variables data matrix. A view of the data is used if the data type matches, otherwise, a copy is made.
obs – Key-indexed one-dimensional observations annotation of length #observations.
var – Key-indexed one-dimensional variables annotation of length #variables.
uns – Key-indexed unstructured annotation.
obsm – Key-indexed multi-dimensional observations annotation of length #observations. If passing a
ndarray
, it needs to have a structured datatype.varm – Key-indexed multi-dimensional variables annotation of length #variables. If passing a
ndarray
, it needs to have a structured datatype.layers – Key-indexed multi-dimensional arrays aligned to dimensions of X.
dtype – Data type used for storage.
shape – Shape tuple (#observations, #variables). Can only be provided if X is None.
filename – Name of backing file. See
h5py.File
.filemode – Open mode of backing file. See
h5py.File
.
See also
read_h5ad
,read_csv
,read_excel
,read_hdf
,read_loom
,read_zarr
,read_mtx
,read_text
,read_umi_tools
Notes
AnnData
stores observations (samples) of variables/features in the rows of a matrix. This is the convention of the modern classics of statistics [Hastie09] and machine learning [Murphy12], the convention of dataframes both in R and Python and the established statistics and machine learning packages in Python (statsmodels_, scikit-learn_).Single dimensional annotations of the observation and variables are stored in the
obs
andvar
attributes asDataFrame
s. This is intended for metrics calculated over their axes. Multi-dimensional annotations are stored inobsm
andvarm
, which are aligned to the objects observation and variable dimensions respectively. Square matrices representing graphs are stored inobsp
andvarp
, with both of their own dimensions aligned to their associated axis. Additional measurements across both observations and variables are stored inlayers
.Indexing into an AnnData object can be performed by relative position with numeric indices (like pandas’
iloc()
), or by labels (likeloc()
). To avoid ambiguity with numeric indexing into observations or variables, indexes of the AnnData object are converted to strings by the constructor.Subsetting an AnnData object by indexing into it will also subset its elements according to the dimensions they were aligned to. This means an operation like adata[list_of_obs, :] will also subset
obs
,obsm
, andlayers
.Subsetting an AnnData object returns a view into the original object, meaning very little additional memory is used upon subsetting. This is achieved lazily, meaning that the constituent arrays are subset on access. Copying a view causes an equivalent “real” AnnData object to be generated. Attempting to modify a view (at any attribute except X) is handled in a copy-on-modify manner, meaning the object is initialized in place. Here’s an example:
batch1 = adata[adata.obs["batch"] == "batch1", :] batch1.obs["value"] = 0 # This makes batch1 a “real” AnnData object
At the end of this snippet: adata was not modified, and batch1 is its own AnnData object with its own data.
Similar to Bioconductor’s ExpressionSet and
scipy.sparse
matrices, subsetting an AnnData object retains the dimensionality of its constituent arrays. Therefore, unlike with the classes exposed bypandas
,numpy
, and xarray, there is no concept of a one dimensional AnnData object. AnnDatas always have two inherent dimensions,obs
andvar
. Additionally, maintaining the dimensionality of the AnnData object allows for consistent handling ofscipy.sparse
matrices andnumpy
arrays.- property T: anndata._core.anndata.AnnData¶
Transpose whole object.
Data matrix is transposed, observations and variables are interchanged.
Ignores .raw.
- Return type
- property X: Optional[Union[numpy.ndarray, scipy.sparse._base.spmatrix, anndata._core.views.ArrayView]]¶
- chunk_X(select=1000, replace=True)[source]¶
Return a chunk of the data matrix
X
with random or specified indices.- Parameters
select – Depending on the type:
replace – If select is an integer then True means random sampling of indices with replacement, False without replacement.
- chunked_X(chunk_size=None)[source]¶
Return an iterator over the rows of the data matrix
X
.- Parameters
chunk_size – Row size of a single chunk.
- concatenate(*adatas, join='inner', batch_key='batch', batch_categories=None, uns_merge=None, index_unique='-', fill_value=None)[source]¶
Concatenate along the observations axis.
The
uns
,varm
andobsm
attributes are ignored.Currently, this works only in ‘memory’ mode.
Note
For more flexible and efficient concatenation, see:
concat()
.- Parameters
adatas – AnnData matrices to concatenate with. Each matrix is referred to as a “batch”.
join – Use intersection (‘inner’) or union (‘outer’) of variables.
batch_key – Add the batch annotation to
obs
using this key.batch_categories – Use these as categories for the batch annotation. By default, use increasing numbers.
uns_merge – Strategy to use for merging entries of uns. These strategies are applied recusivley. Currently implemented strategies include:
None: The default. The concatenated object will just have an empty dict for uns.
“same”: Only entries which have the same value in all AnnData objects are kept.
“unique”: Only entries which have one unique value in all AnnData objects are kept.
“first”: The first non-missing value is used.
“only”: A value is included if only one of the AnnData objects has a value at this path.
index_unique – Make the index unique by joining the existing index names with the batch category, using index_unique=’-’, for instance. Provide None to keep existing indices.
fill_value – Scalar value to fill newly missing values in arrays with. Note: only applies to arrays and sparse matrices (not dataframes) and will only be used if join=”outer”.
Note
If not provided, the default value is 0 for sparse matrices and np.nan for numpy arrays. See the examples below for more information.
- Returns
The concatenated
AnnData
, where adata.obs[batch_key] stores a categorical variable labeling the batch.- Return type
AnnData
Notes
Warning
If you use join=’outer’ this fills 0s for sparse data when variables are absent in a batch. Use this with care. Dense data is filled with NaN. See the examples.
Examples
Joining on intersection of variables.
>>> adata1 = AnnData( ... np.array([[1, 2, 3], [4, 5, 6]]), ... dict(obs_names=['s1', 's2'], anno1=['c1', 'c2']), ... dict(var_names=['a', 'b', 'c'], annoA=[0, 1, 2]), ... ) >>> adata2 = AnnData( ... np.array([[1, 2, 3], [4, 5, 6]]), ... dict(obs_names=['s3', 's4'], anno1=['c3', 'c4']), ... dict(var_names=['d', 'c', 'b'], annoA=[0, 1, 2]), ... ) >>> adata3 = AnnData( ... np.array([[1, 2, 3], [4, 5, 6]]), ... dict(obs_names=['s1', 's2'], anno2=['d3', 'd4']), ... dict(var_names=['d', 'c', 'b'], annoA=[0, 2, 3], annoB=[0, 1, 2]), ... ) >>> adata = adata1.concatenate(adata2, adata3) >>> adata AnnData object with n_obs × n_vars = 6 × 2 obs: 'anno1', 'anno2', 'batch' var: 'annoA-0', 'annoA-1', 'annoA-2', 'annoB-2' >>> adata.X array([[2., 3.], [5., 6.], [3., 2.], [6., 5.], [3., 2.], [6., 5.]], dtype=float32) >>> adata.obs anno1 anno2 batch s1-0 c1 NaN 0 s2-0 c2 NaN 0 s3-1 c3 NaN 1 s4-1 c4 NaN 1 s1-2 NaN d3 2 s2-2 NaN d4 2 >>> adata.var.T b c annoA-0 1 2 annoA-1 2 1 annoA-2 3 2 annoB-2 2 1
Joining on the union of variables.
>>> outer = adata1.concatenate(adata2, adata3, join='outer') >>> outer AnnData object with n_obs × n_vars = 6 × 4 obs: 'anno1', 'anno2', 'batch' var: 'annoA-0', 'annoA-1', 'annoA-2', 'annoB-2' >>> outer.var.T a b c d annoA-0 0.0 1.0 2.0 NaN annoA-1 NaN 2.0 1.0 0.0 annoA-2 NaN 3.0 2.0 0.0 annoB-2 NaN 2.0 1.0 0.0 >>> outer.var_names Index(['a', 'b', 'c', 'd'], dtype='object') >>> outer.X array([[ 1., 2., 3., nan], [ 4., 5., 6., nan], [nan, 3., 2., 1.], [nan, 6., 5., 4.], [nan, 3., 2., 1.], [nan, 6., 5., 4.]], dtype=float32) >>> outer.X.sum(axis=0) array([nan, 25., 23., nan], dtype=float32) >>> import pandas as pd >>> Xdf = pd.DataFrame(outer.X, columns=outer.var_names) >>> Xdf a b c d 0 1.0 2.0 3.0 NaN 1 4.0 5.0 6.0 NaN 2 NaN 3.0 2.0 1.0 3 NaN 6.0 5.0 4.0 4 NaN 3.0 2.0 1.0 5 NaN 6.0 5.0 4.0 >>> Xdf.sum() a 5.0 b 25.0 c 23.0 d 10.0 dtype: float32
One way to deal with missing values is to use masked arrays:
>>> from numpy import ma >>> outer.X = ma.masked_invalid(outer.X) >>> outer.X masked_array( data=[[1.0, 2.0, 3.0, --], [4.0, 5.0, 6.0, --], [--, 3.0, 2.0, 1.0], [--, 6.0, 5.0, 4.0], [--, 3.0, 2.0, 1.0], [--, 6.0, 5.0, 4.0]], mask=[[False, False, False, True], [False, False, False, True], [ True, False, False, False], [ True, False, False, False], [ True, False, False, False], [ True, False, False, False]], fill_value=1e+20, dtype=float32) >>> outer.X.sum(axis=0).data array([ 5., 25., 23., 10.], dtype=float32)
The masked array is not saved but has to be reinstantiated after saving.
>>> outer.write('./test.h5ad') >>> from anndata import read_h5ad >>> outer = read_h5ad('./test.h5ad') >>> outer.X array([[ 1., 2., 3., nan], [ 4., 5., 6., nan], [nan, 3., 2., 1.], [nan, 6., 5., 4.], [nan, 3., 2., 1.], [nan, 6., 5., 4.]], dtype=float32)
For sparse data, everything behaves similarly, except that for join=’outer’, zeros are added.
>>> from scipy.sparse import csr_matrix >>> adata1 = AnnData( ... csr_matrix([[0, 2, 3], [0, 5, 6]]), ... dict(obs_names=['s1', 's2'], anno1=['c1', 'c2']), ... dict(var_names=['a', 'b', 'c']), ... ) >>> adata2 = AnnData( ... csr_matrix([[0, 2, 3], [0, 5, 6]]), ... dict(obs_names=['s3', 's4'], anno1=['c3', 'c4']), ... dict(var_names=['d', 'c', 'b']), ... ) >>> adata3 = AnnData( ... csr_matrix([[1, 2, 0], [0, 5, 6]]), ... dict(obs_names=['s5', 's6'], anno2=['d3', 'd4']), ... dict(var_names=['d', 'c', 'b']), ... ) >>> adata = adata1.concatenate(adata2, adata3, join='outer') >>> adata.var_names Index(['a', 'b', 'c', 'd'], dtype='object') >>> adata.X.toarray() array([[0., 2., 3., 0.], [0., 5., 6., 0.], [0., 3., 2., 0.], [0., 6., 5., 0.], [0., 0., 2., 1.], [0., 6., 5., 0.]], dtype=float32)
- property filename: Optional[pathlib.Path]¶
Change to backing mode by setting the filename of a .h5ad file.
Setting the filename writes the stored data to disk.
Setting the filename when the filename was previously another name moves the backing file from the previous file to the new file. If you want to copy the previous file, use copy(filename=’new_filename’).
- property is_view: bool¶
True if object is view of another AnnData object, False otherwise.
- Return type
- property layers: Union[anndata._core.aligned_mapping.Layers, anndata._core.aligned_mapping.LayersView]¶
Dictionary-like object with values of the same dimensions as
X
.Layers in AnnData are inspired by loompy’s loomlayers.
Return the layer named “unspliced”:
adata.layers["unspliced"]
Create or replace the “spliced” layer:
adata.layers["spliced"] = ...
Assign the 10th column of layer “spliced” to the variable a:
a = adata.layers["spliced"][:, 10]
Delete the “spliced” layer:
del adata.layers["spliced"]
Return layers’ names:
adata.layers.keys()
- Return type
Union
[Layers
,LayersView
]
- property obs: pandas.core.frame.DataFrame¶
One-dimensional annotation of observations (pd.DataFrame).
- Return type
- property obs_names: pandas.core.indexes.base.Index¶
Names of observations (alias for .obs.index).
- Return type
- obs_names_make_unique(join='-')[source]¶
Makes the index unique by appending a number string to each duplicate index element: ‘1’, ‘2’, etc.
If a tentative name created by the algorithm already exists in the index, it tries the next integer in the sequence.
The first occurrence of a non-unique value is ignored.
- Parameters
join – The connecting string between name and integer.
Examples
>>> from anndata import AnnData >>> adata = AnnData(np.ones((2, 3)), var=pd.DataFrame(index=["a", "a", "b"])) >>> adata.var_names Index(['a', 'a', 'b'], dtype='object') >>> adata.var_names_make_unique() >>> adata.var_names Index(['a', 'a-1', 'b'], dtype='object')
- obs_vector(k, *, layer=None)[source]¶
Convenience function for returning a 1 dimensional ndarray of values from
X
,layers
[k], orobs
.Made for convenience, not performance. Intentionally permissive about arguments, for easy iterative use.
- k
- layer
What layer values should be returned from. If None,
X
is used.
- property obsm: Union[anndata._core.aligned_mapping.AxisArrays, anndata._core.aligned_mapping.AxisArraysView]¶
Multi-dimensional annotation of observations (mutable structured
ndarray
).Stores for each key a two or higher-dimensional
ndarray
of length n_obs. Is sliced with data and obs but behaves otherwise like a mapping.- Return type
Union
[AxisArrays
,AxisArraysView
]
- property obsp: Union[anndata._core.aligned_mapping.PairwiseArrays, anndata._core.aligned_mapping.PairwiseArraysView]¶
Pairwise annotation of observations, a mutable mapping with array-like values.
Stores for each key a two or higher-dimensional
ndarray
whose first two dimensions are of length n_obs. Is sliced with data and obs but behaves otherwise like a mapping.- Return type
Union
[PairwiseArrays
,PairwiseArraysView
]
- property raw: anndata._core.raw.Raw¶
Store raw version of
X
andvar
as .raw.X and .raw.var.The
raw
attribute is initialized with the current content of an object by setting:adata.raw = adata
Its content can be deleted:
adata.raw = None # or del adata.raw
Upon slicing an AnnData object along the obs (row) axis,
raw
is also sliced. Slicing an AnnData object along the vars (columns) axis leavesraw
unaffected. Note that you can call:adata.raw[:, 'orig_variable_name'].X
to retrieve the data associated with a variable that might have been filtered out or “compressed away” in
X
.- Return type
Raw
- rename_categories(key, categories)[source]¶
Rename categories of annotation key in
obs
,var
, anduns
.Only supports passing a list/array-like categories argument.
Besides calling self.obs[key].cat.categories = categories – similar for
var
- this also renames categories in unstructured annotation that uses the categorical annotation key.- Parameters
key – Key for observations or variables annotation.
categories – New categories, the same number as the old categories.
- strings_to_categoricals(df=None)[source]¶
Transform string annotations to categoricals.
Only affects string annotations that lead to less categories than the total number of observations.
Notes
Turns the view of an
AnnData
into an actualAnnData
.
- to_df(layer=None)[source]¶
Generate shallow
DataFrame
.The data matrix
X
is returned asDataFrame
, whereobs_names
initializes the index, andvar_names
the columns.No annotations are maintained in the returned object.
The data matrix is densified in case it is sparse.
- layerstr
Key for .layers.
- Return type
- to_memory()[source]¶
Load backed AnnData object into memory.
Example
import anndata backed = anndata.read_h5ad("file.h5ad", backed="r") mem = backed[backed.obs["cluster"] == "a", :].to_memory()
- Return type
- transpose()[source]¶
Transpose whole object.
Data matrix is transposed, observations and variables are interchanged.
Ignores .raw.
- Return type
- property uns: MutableMapping¶
Unstructured annotation (ordered dictionary).
- Return type
- property var: pandas.core.frame.DataFrame¶
One-dimensional annotation of variables/ features (pd.DataFrame).
- Return type
- property var_names: pandas.core.indexes.base.Index¶
Names of variables (alias for .var.index).
- Return type
- var_names_make_unique(join='-')[source]¶
Makes the index unique by appending a number string to each duplicate index element: ‘1’, ‘2’, etc.
If a tentative name created by the algorithm already exists in the index, it tries the next integer in the sequence.
The first occurrence of a non-unique value is ignored.
- Parameters
join – The connecting string between name and integer.
Examples
>>> from anndata import AnnData >>> adata = AnnData(np.ones((2, 3)), var=pd.DataFrame(index=["a", "a", "b"])) >>> adata.var_names Index(['a', 'a', 'b'], dtype='object') >>> adata.var_names_make_unique() >>> adata.var_names Index(['a', 'a-1', 'b'], dtype='object')
- var_vector(k, *, layer=None)[source]¶
Convenience function for returning a 1 dimensional ndarray of values from
X
,layers
[k], orobs
.Made for convenience, not performance. Intentionally permissive about arguments, for easy iterative use.
- k
- layer
What layer values should be returned from. If None,
X
is used.
- property varm: Union[anndata._core.aligned_mapping.AxisArrays, anndata._core.aligned_mapping.AxisArraysView]¶
Multi-dimensional annotation of variables/features (mutable structured
ndarray
).Stores for each key a two or higher-dimensional
ndarray
of length n_vars. Is sliced with data and var but behaves otherwise like a mapping.- Return type
Union
[AxisArrays
,AxisArraysView
]
- property varp: Union[anndata._core.aligned_mapping.PairwiseArrays, anndata._core.aligned_mapping.PairwiseArraysView]¶
Pairwise annotation of observations, a mutable mapping with array-like values.
Stores for each key a two or higher-dimensional
ndarray
whose first two dimensions are of length n_var. Is sliced with data and var but behaves otherwise like a mapping.- Return type
Union
[PairwiseArrays
,PairwiseArraysView
]
- write(filename=None, compression=None, compression_opts=None, force_dense=None, as_dense=())¶
Write .h5ad-formatted hdf5 file.
Note
Setting compression to ‘gzip’ can save disk space but will slow down writing and subsequent reading. Prior to v0.6.16, this was the default for parameter compression.
Generally, if you have sparse data that are stored as a dense matrix, you can dramatically improve performance and reduce disk space by converting to a
csr_matrix
:from scipy.sparse import csr_matrix adata.X = csr_matrix(adata.X)
- Parameters
filename – Filename of data file. Defaults to backing file.
compression – See the h5py dataset_compression.
compression_opts – See the h5py dataset_compression.
as_dense – Sparse arrays in AnnData object to write as dense. Currently only supports X and raw/X.
force_dense – Write sparse data as a dense matrix. Defaults to True if object is backed, otherwise to False.
- write_csvs(dirname, skip_data=True, sep=',')[source]¶
Write annotation to .csv files.
It is not possible to recover the full
AnnData
from these files. Usewrite()
for this.- Parameters
dirname – Name of directory to which to export.
skip_data – Skip the data matrix
X
.sep – Separator for the data.
- write_h5ad(filename=None, compression=None, compression_opts=None, force_dense=None, as_dense=())[source]¶
Write .h5ad-formatted hdf5 file.
Note
Setting compression to ‘gzip’ can save disk space but will slow down writing and subsequent reading. Prior to v0.6.16, this was the default for parameter compression.
Generally, if you have sparse data that are stored as a dense matrix, you can dramatically improve performance and reduce disk space by converting to a
csr_matrix
:from scipy.sparse import csr_matrix adata.X = csr_matrix(adata.X)
- Parameters
filename – Filename of data file. Defaults to backing file.
compression – See the h5py dataset_compression.
compression_opts – See the h5py dataset_compression.
as_dense – Sparse arrays in AnnData object to write as dense. Currently only supports X and raw/X.
force_dense – Write sparse data as a dense matrix. Defaults to True if object is backed, otherwise to False.
- write_loom(filename, write_obsm_varm=False)[source]¶
Write .loom-formatted hdf5 file.
- Parameters
filename – The filename.
- write_zarr(store, chunks=None)[source]¶
Write a hierarchical Zarr array store.
- Parameters
store – The filename, a
MutableMapping
, or a Zarr storage class.chunks – Chunk shape.
- imc.types.Array¶
alias of
numpy.ndarray
- class imc.types.Axis(axes, pickradius=15)[source]¶
Base class for .XAxis and .YAxis.
- Variables
~Axis.isDefault_label (bool) –
~Axis.axes (matplotlib.axes.Axes) – The ~.axes.Axes to which the Axis belongs.
~Axis.major (matplotlib.axis.Ticker) – Determines the major tick positions and their label format.
~Axis.minor (matplotlib.axis.Ticker) – Determines the minor tick positions and their label format.
~Axis.callbacks (matplotlib.cbook.CallbackRegistry) –
~Axis.label (.Text) – The axis label.
~Axis.labelpad (float) – The distance between the axis label and the tick labels. Defaults to :rc:`axes.labelpad` = 4.
~Axis.offsetText (.Text) – A .Text object containing the data offset of the ticks (if any).
~Axis.pickradius (float) – The acceptance radius for containment tests. See also .Axis.contains.
~Axis.majorTicks (list of .Tick) – The major ticks.
~Axis.minorTicks (list of .Tick) – The minor ticks.
- axis_date(tz=None)[source]¶
Set up axis ticks and labels to treat data along this Axis as dates.
- Parameters
tz (str or datetime.tzinfo, default: :rc:`timezone`) – The timezone used to create date labels.
- clear()[source]¶
Clear the axis.
This resets axis properties to their default values:
the label
the scale
locators, formatters and ticks
major and minor grid
units
registered callbacks
- draw(renderer, *args, **kwargs)[source]¶
Draw the Artist (and its children) using the given renderer.
This has no effect if the artist is not visible (.Artist.get_visible returns False).
- Parameters
renderer (.RendererBase subclass.)
Notes
This method is overridden in the Artist subclasses.
- get_inverted()[source]¶
Return whether this Axis is oriented in the “inverse” direction.
The “normal” direction is increasing to the right for the x-axis and to the top for the y-axis; the “inverse” direction is increasing to the left for the x-axis and to the bottom for the y-axis.
- get_ticklabel_extents(renderer)[source]¶
Get the extents of the tick labels on either side of the axes.
- get_ticklabels(minor=False, which=None)[source]¶
Get this Axis’ tick labels.
- Parameters
minor (bool) – Whether to return the minor or the major ticklabels.
which (None, (‘minor’, ‘major’, ‘both’)) – Overrides minor.
Selects which ticklabels to return
- Return type
list of ~matplotlib.text.Text
Notes
The tick label strings are not populated until a
draw
method has been called.See also: ~.pyplot.draw and ~.FigureCanvasBase.draw.
- get_ticks_direction(minor=False)[source]¶
Get the tick directions as a numpy array
- Parameters
minor (bool, default: False) – True to return the minor tick directions, False to return the major tick directions.
- Return type
numpy array of tick directions
- get_tightbbox(renderer, *, for_layout_only=False)[source]¶
Return a bounding box that encloses the axis. It only accounts tick labels, axis label, and offsetText.
If for_layout_only is True, then the width of the label (if this is an x-axis) or the height of the label (if this is a y-axis) is collapsed to near zero. This allows tight/constrained_layout to ignore too-long labels when doing their layout.
- grid(visible=None, which='major', **kwargs)[source]¶
Configure the grid lines.
- Parameters
visible (bool or None) – Whether to show the grid lines. If any kwargs are supplied, it is assumed you want the grid on and visible will be set to True.
If visible is None and there are no kwargs, this toggles the visibility of the lines.
which ({‘major’, ‘minor’, ‘both’}) – The grid lines to apply the changes on.
**kwargs (.Line2D properties) –
Define the line properties of the grid, e.g.:
grid(color='r', linestyle='-', linewidth=2)
- property remove_overlapping_locs¶
If minor ticker locations that overlap with major ticker locations should be trimmed.
- reset_ticks()[source]¶
Re-initialize the major and minor Tick lists.
Each list starts with a single fresh Tick.
- set(*, agg_filter=<UNSET>, alpha=<UNSET>, animated=<UNSET>, clip_box=<UNSET>, clip_on=<UNSET>, clip_path=<UNSET>, data_interval=<UNSET>, gid=<UNSET>, in_layout=<UNSET>, inverted=<UNSET>, label=<UNSET>, label_coords=<UNSET>, label_position=<UNSET>, label_text=<UNSET>, major_formatter=<UNSET>, major_locator=<UNSET>, minor_formatter=<UNSET>, minor_locator=<UNSET>, path_effects=<UNSET>, picker=<UNSET>, pickradius=<UNSET>, rasterized=<UNSET>, remove_overlapping_locs=<UNSET>, sketch_params=<UNSET>, snap=<UNSET>, tick_params=<UNSET>, ticklabels=<UNSET>, ticks=<UNSET>, transform=<UNSET>, units=<UNSET>, url=<UNSET>, view_interval=<UNSET>, visible=<UNSET>, zorder=<UNSET>)¶
Set multiple properties at once.
Supported properties are
- Properties:
agg_filter: a filter function, which takes a (m, n, 3) float array and a dpi value, and returns a (m, n, 3) array alpha: scalar or None animated: bool clip_box: .Bbox clip_on: bool clip_path: unknown data_interval: unknown figure: .Figure gid: str in_layout: bool inverted: unknown label: object label_coords: unknown label_position: {‘top’, ‘bottom’} label_text: str major_formatter: ~matplotlib.ticker.Formatter,
str
, or function major_locator: ~matplotlib.ticker.Locator minor_formatter: ~matplotlib.ticker.Formatter,str
, or function minor_locator: ~matplotlib.ticker.Locator path_effects: .AbstractPathEffect picker: None or bool or float or callable pickradius: float rasterized: bool remove_overlapping_locs: unknown sketch_params: (scale: float, length: float, randomness: float) snap: bool or None tick_params: unknown ticklabels: sequence of str or of .Texts ticks: list of floats transform: .Transform units: units tag url: str view_interval: unknown visible: bool zorder: float
- set_clip_path(clippath, transform=None)[source]¶
Set the artist’s clip path.
- Parameters
path (.Patch or .Path or .TransformedPath or None) – The clip path. If given a .Path, transform must be provided as well. If None, a previously set clip path is removed.
transform (~matplotlib.transforms.Transform, optional) – Only used if path is a .Path, in which case the given .Path is converted to a .TransformedPath using transform.
Notes
For efficiency, if path is a .Rectangle this method will set the clipping box to the corresponding rectangle and set the clipping path to
None
.For technical reasons (support of ~.Artist.set), a tuple (path, transform) is also accepted as a single positional parameter.
- set_data_interval(vmin, vmax, ignore=False)[source]¶
Set the axis data limits. This method is for internal use.
If ignore is False (the default), this method will never reduce the preexisting data limits, only expand them if vmin or vmax are not within them. Moreover, the order of vmin and vmax does not matter; the orientation of the axis will not change.
If ignore is True, the data limits will be set exactly to
(vmin, vmax)
in that order.
- set_default_intervals()[source]¶
Set the default limits for the axis data and view interval if they have not been not mutated yet.
- set_inverted(inverted)[source]¶
Set whether this Axis is oriented in the “inverse” direction.
The “normal” direction is increasing to the right for the x-axis and to the top for the y-axis; the “inverse” direction is increasing to the left for the x-axis and to the bottom for the y-axis.
- set_label_coords(x, y, transform=None)[source]¶
Set the coordinates of the label.
By default, the x coordinate of the y label and the y coordinate of the x label are determined by the tick label bounding boxes, but this can lead to poor alignment of multiple labels if there are multiple axes.
You can also specify the coordinate system of the label with the transform. If None, the default coordinate system will be the axes coordinate system: (0, 0) is bottom left, (0.5, 0.5) is center, etc.
- set_label_position(position)[source]¶
Set the label position (top or bottom)
- Parameters
position ({‘top’, ‘bottom’})
- set_label_text(label, fontdict=None, **kwargs)[source]¶
Set the text value of the axis label.
- Parameters
label (str) – Text string.
fontdict (dict) – Text properties.
**kwargs – Merged into fontdict.
- set_major_formatter(formatter)[source]¶
Set the formatter of the major ticker.
In addition to a ~matplotlib.ticker.Formatter instance, this also accepts a
str
or function.For a
str
a ~matplotlib.ticker.StrMethodFormatter is used. The field used for the value must be labeled'x'
and the field used for the position must be labeled'pos'
. See the ~matplotlib.ticker.StrMethodFormatter documentation for more information.For a function, a ~matplotlib.ticker.FuncFormatter is used. The function must take two inputs (a tick value
x
and a positionpos
), and return a string containing the corresponding tick label. See the ~matplotlib.ticker.FuncFormatter documentation for more information.- Parameters
formatter (~matplotlib.ticker.Formatter,
str
, or function)
- set_major_locator(locator)[source]¶
Set the locator of the major ticker.
- Parameters
locator (~matplotlib.ticker.Locator)
- set_minor_formatter(formatter)[source]¶
Set the formatter of the minor ticker.
In addition to a ~matplotlib.ticker.Formatter instance, this also accepts a
str
or function. See .Axis.set_major_formatter for more information.- Parameters
formatter (~matplotlib.ticker.Formatter,
str
, or function)
- set_minor_locator(locator)[source]¶
Set the locator of the minor ticker.
- Parameters
locator (~matplotlib.ticker.Locator)
- set_pickradius(pickradius)[source]¶
Set the depth of the axis used by the picker.
- Parameters
pickradius (float)
- set_tick_params(which='major', reset=False, **kw)[source]¶
Set appearance parameters for ticks, ticklabels, and gridlines.
For documentation of keyword arguments, see
matplotlib.axes.Axes.tick_params()
.
- set_ticklabels(ticklabels, *, minor=False, **kwargs)[source]¶
Set the text values of the tick labels.
Discouraged
The use of this method is discouraged, because of the dependency on tick positions. In most cases, you’ll want to use
set_[x/y]ticks(positions, labels)
instead.If you are using this method, you should always fix the tick positions before, e.g. by using .Axis.set_ticks or by explicitly setting a ~.ticker.FixedLocator. Otherwise, ticks are free to move and the labels may end up in unexpected positions.
- Parameters
ticklabels (sequence of str or of .Texts) – Texts for labeling each tick location in the sequence set by .Axis.set_ticks; the number of labels must match the number of locations.
minor (bool) – If True, set minor ticks instead of major ticks.
**kwargs – Text properties.
- Returns
For each tick, includes
tick.label1
if it is visible, thentick.label2
if it is visible, in that order.- Return type
list of .Texts
- set_ticks(ticks, labels=None, *, minor=False, **kwargs)[source]¶
Set this Axis’ tick locations and optionally labels.
If necessary, the view limits of the Axis are expanded so that all given ticks are visible.
- Parameters
ticks (list of floats) – List of tick locations.
labels (list of str, optional) – List of tick labels. If not set, the labels show the data value.
minor (bool, default: False) – If
False
, set the major ticks; ifTrue
, the minor ticks.**kwargs – .Text properties for the labels. These take effect only if you pass labels. In other cases, please use ~.Axes.tick_params.
Notes
The mandatory expansion of the view limits is an intentional design choice to prevent the surprise of a non-visible tick. If you need other limits, you should set the limits explicitly after setting the ticks.
- set_units(u)[source]¶
Set the units for axis.
- Parameters
u (units tag)
Notes
The units of any shared axis will also be updated.
- set_view_interval(vmin, vmax, ignore=False)[source]¶
Set the axis view limits. This method is for internal use; Matplotlib users should typically use e.g. ~.Axes.set_xlim or ~.Axes.set_ylim.
If ignore is False (the default), this method will never reduce the preexisting view limits, only expand them if vmin or vmax are not within them. Moreover, the order of vmin and vmax does not matter; the orientation of the axis will not change.
If ignore is True, the view limits will be set exactly to
(vmin, vmax)
in that order.
- imc.types.ColorMap¶
alias of
matplotlib.colors.LinearSegmentedColormap
- class imc.types.DataFrame(data=None, index=None, columns=None, dtype=None, copy=None)[source]¶
Two-dimensional, size-mutable, potentially heterogeneous tabular data.
Data structure also contains labeled axes (rows and columns). Arithmetic operations align on both row and column labels. Can be thought of as a dict-like container for Series objects. The primary pandas data structure.
- Parameters
data (ndarray (structured or homogeneous), Iterable, dict, or DataFrame) – Dict can contain Series, arrays, constants, dataclass or list-like objects. If data is a dict, column order follows insertion-order. If a dict contains Series which have an index defined, it is aligned by its index.
Changed in version 0.25.0: If data is a list of dicts, column order follows insertion-order.
index (Index or array-like) – Index to use for resulting frame. Will default to RangeIndex if no indexing information part of input data and no index provided.
columns (Index or array-like) – Column labels to use for resulting frame when data does not have them, defaulting to RangeIndex(0, 1, 2, …, n). If data contains column labels, will perform column selection instead.
dtype (dtype, default None) – Data type to force. Only a single dtype is allowed. If None, infer.
copy (bool or None, default None) – Copy data from inputs. For dict data, the default of None behaves like
copy=True
. For DataFrame or 2d ndarray input, the default of None behaves likecopy=False
.Changed in version 1.3.0.
See also
DataFrame.from_records
Constructor from tuples, also record arrays.
DataFrame.from_dict
From dicts of Series, arrays, or dicts.
read_csv
Read a comma-separated values (csv) file into DataFrame.
read_table
Read general delimited file into DataFrame.
read_clipboard
Read text from clipboard into DataFrame.
Examples
Constructing DataFrame from a dictionary.
>>> d = {'col1': [1, 2], 'col2': [3, 4]} >>> df = pd.DataFrame(data=d) >>> df col1 col2 0 1 3 1 2 4
Notice that the inferred dtype is int64.
>>> df.dtypes col1 int64 col2 int64 dtype: object
To enforce a single dtype:
>>> df = pd.DataFrame(data=d, dtype=np.int8) >>> df.dtypes col1 int8 col2 int8 dtype: object
Constructing DataFrame from a dictionary including Series:
>>> d = {'col1': [0, 1, 2, 3], 'col2': pd.Series([2, 3], index=[2, 3])} >>> pd.DataFrame(data=d, index=[0, 1, 2, 3]) col1 col2 0 0 NaN 1 1 NaN 2 2 2.0 3 3 3.0
Constructing DataFrame from numpy ndarray:
>>> df2 = pd.DataFrame(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]), ... columns=['a', 'b', 'c']) >>> df2 a b c 0 1 2 3 1 4 5 6 2 7 8 9
Constructing DataFrame from a numpy ndarray that has labeled columns:
>>> data = np.array([(1, 2, 3), (4, 5, 6), (7, 8, 9)], ... dtype=[("a", "i4"), ("b", "i4"), ("c", "i4")]) >>> df3 = pd.DataFrame(data, columns=['c', 'a']) ... >>> df3 c a 0 3 1 1 6 4 2 9 7
Constructing DataFrame from dataclass:
>>> from dataclasses import make_dataclass >>> Point = make_dataclass("Point", [("x", int), ("y", int)]) >>> pd.DataFrame([Point(0, 0), Point(0, 3), Point(2, 3)]) x y 0 0 0 1 0 3 2 2 3
- add(other, axis='columns', level=None, fill_value=None)¶
Get Addition of dataframe and other, element-wise (binary operator add).
Equivalent to
dataframe + other
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, radd.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- agg(func=None, axis=0, *args, **kwargs)¶
Aggregate using one or more operations over the specified axis.
- Parameters
func (function, str, list or dict) – Function to use for aggregating the data. If a function, must either work when passed a DataFrame or when passed to DataFrame.apply.
Accepted combinations are:
function
string function name
list of functions and/or function names, e.g.
[np.sum, 'mean']
dict of axis labels -> functions, function names or list of such.
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – If 0 or ‘index’: apply function to each column. If 1 or ‘columns’: apply function to each row.
*args – Positional arguments to pass to func.
**kwargs – Keyword arguments to pass to func.
- Returns
scalar, Series or DataFrame – The return can be:
scalar : when Series.agg is called with single function
Series : when DataFrame.agg is called with a single function
DataFrame : when DataFrame.agg is called with several functions
Return scalar, Series or DataFrame.
The aggregation operations are always performed over an axis, either the
index (default) or the column axis. This behavior is different from
numpy aggregation functions (mean, median, prod, sum, std,
var), where the default is to compute the aggregation of the flattened
array, e.g.,
numpy.mean(arr_2d)
as opposed tonumpy.mean(arr_2d, axis=0)
.agg is an alias for aggregate. Use the alias.
See also
DataFrame.apply
Perform any type of operations.
DataFrame.transform
Perform transformation type operations.
core.groupby.GroupBy
Perform operations over groups.
core.resample.Resampler
Perform operations over resampled bins.
core.window.Rolling
Perform operations over rolling window.
core.window.Expanding
Perform operations over expanding window.
core.window.ExponentialMovingWindow
Perform operation over exponential weighted window.
Notes
agg is an alias for aggregate. Use the alias.
Functions that mutate the passed object can produce unexpected behavior or errors and are not supported. See Mutating with User Defined Function (UDF) methods for more details.
A passed user-defined-function will be passed a Series for evaluation.
Examples
>>> df = pd.DataFrame([[1, 2, 3], ... [4, 5, 6], ... [7, 8, 9], ... [np.nan, np.nan, np.nan]], ... columns=['A', 'B', 'C'])
Aggregate these functions over the rows.
>>> df.agg(['sum', 'min']) A B C sum 12.0 15.0 18.0 min 1.0 2.0 3.0
Different aggregations per column.
>>> df.agg({'A' : ['sum', 'min'], 'B' : ['min', 'max']}) A B sum 12.0 NaN min 1.0 2.0 max NaN 8.0
Aggregate different functions over the columns and rename the index of the resulting DataFrame.
>>> df.agg(x=('A', max), y=('B', 'min'), z=('C', np.mean)) A B C x 7.0 NaN NaN y NaN 2.0 NaN z NaN NaN 6.0
Aggregate over the columns.
>>> df.agg("mean", axis="columns") 0 2.0 1 5.0 2 8.0 3 NaN dtype: float64
- aggregate(func=None, axis=0, *args, **kwargs)[source]¶
Aggregate using one or more operations over the specified axis.
- Parameters
func (function, str, list or dict) – Function to use for aggregating the data. If a function, must either work when passed a DataFrame or when passed to DataFrame.apply.
Accepted combinations are:
function
string function name
list of functions and/or function names, e.g.
[np.sum, 'mean']
dict of axis labels -> functions, function names or list of such.
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – If 0 or ‘index’: apply function to each column. If 1 or ‘columns’: apply function to each row.
*args – Positional arguments to pass to func.
**kwargs – Keyword arguments to pass to func.
- Returns
scalar, Series or DataFrame – The return can be:
scalar : when Series.agg is called with single function
Series : when DataFrame.agg is called with a single function
DataFrame : when DataFrame.agg is called with several functions
Return scalar, Series or DataFrame.
The aggregation operations are always performed over an axis, either the
index (default) or the column axis. This behavior is different from
numpy aggregation functions (mean, median, prod, sum, std,
var), where the default is to compute the aggregation of the flattened
array, e.g.,
numpy.mean(arr_2d)
as opposed tonumpy.mean(arr_2d, axis=0)
.agg is an alias for aggregate. Use the alias.
See also
DataFrame.apply
Perform any type of operations.
DataFrame.transform
Perform transformation type operations.
core.groupby.GroupBy
Perform operations over groups.
core.resample.Resampler
Perform operations over resampled bins.
core.window.Rolling
Perform operations over rolling window.
core.window.Expanding
Perform operations over expanding window.
core.window.ExponentialMovingWindow
Perform operation over exponential weighted window.
Notes
agg is an alias for aggregate. Use the alias.
Functions that mutate the passed object can produce unexpected behavior or errors and are not supported. See Mutating with User Defined Function (UDF) methods for more details.
A passed user-defined-function will be passed a Series for evaluation.
Examples
>>> df = pd.DataFrame([[1, 2, 3], ... [4, 5, 6], ... [7, 8, 9], ... [np.nan, np.nan, np.nan]], ... columns=['A', 'B', 'C'])
Aggregate these functions over the rows.
>>> df.agg(['sum', 'min']) A B C sum 12.0 15.0 18.0 min 1.0 2.0 3.0
Different aggregations per column.
>>> df.agg({'A' : ['sum', 'min'], 'B' : ['min', 'max']}) A B sum 12.0 NaN min 1.0 2.0 max NaN 8.0
Aggregate different functions over the columns and rename the index of the resulting DataFrame.
>>> df.agg(x=('A', max), y=('B', 'min'), z=('C', np.mean)) A B C x 7.0 NaN NaN y NaN 2.0 NaN z NaN NaN 6.0
Aggregate over the columns.
>>> df.agg("mean", axis="columns") 0 2.0 1 5.0 2 8.0 3 NaN dtype: float64
- align(other, join='outer', axis=None, level=None, copy=True, fill_value=None, method=None, limit=None, fill_axis=0, broadcast_axis=None)[source]¶
Align two objects on their axes with the specified join method.
Join method is specified for each axis Index.
- Parameters
other (DataFrame or Series)
join ({‘outer’, ‘inner’, ‘left’, ‘right’}, default ‘outer’)
axis (allowed axis of the other object, default None) – Align on index (0), columns (1), or both (None).
level (int or level name, default None) – Broadcast across a level, matching Index values on the passed MultiIndex level.
copy (bool, default True) – Always returns new objects. If copy=False and no reindexing is required then original objects are returned.
fill_value (scalar, default np.NaN) – Value to use for missing values. Defaults to NaN, but can be any “compatible” value.
method ({‘backfill’, ‘bfill’, ‘pad’, ‘ffill’, None}, default None) – Method to use for filling holes in reindexed Series:
pad / ffill: propagate last valid observation forward to next valid.
backfill / bfill: use NEXT valid observation to fill gap.
limit (int, default None) – If method is specified, this is the maximum number of consecutive NaN values to forward/backward fill. In other words, if there is a gap with more than this number of consecutive NaNs, it will only be partially filled. If method is not specified, this is the maximum number of entries along the entire axis where NaNs will be filled. Must be greater than 0 if not None.
fill_axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – Filling axis, method and limit.
broadcast_axis ({0 or ‘index’, 1 or ‘columns’}, default None) – Broadcast values along this axis, if aligning two objects of different dimensions.
- Returns
(left, right) – Aligned objects.
- Return type
(DataFrame, type of other)
Examples
>>> df = pd.DataFrame( ... [[1, 2, 3, 4], [6, 7, 8, 9]], columns=["D", "B", "E", "A"], index=[1, 2] ... ) >>> other = pd.DataFrame( ... [[10, 20, 30, 40], [60, 70, 80, 90], [600, 700, 800, 900]], ... columns=["A", "B", "C", "D"], ... index=[2, 3, 4], ... ) >>> df D B E A 1 1 2 3 4 2 6 7 8 9 >>> other A B C D 2 10 20 30 40 3 60 70 80 90 4 600 700 800 900
Align on columns:
>>> left, right = df.align(other, join="outer", axis=1) >>> left A B C D E 1 4 2 NaN 1 3 2 9 7 NaN 6 8 >>> right A B C D E 2 10 20 30 40 NaN 3 60 70 80 90 NaN 4 600 700 800 900 NaN
We can also align on the index:
>>> left, right = df.align(other, join="outer", axis=0) >>> left D B E A 1 1.0 2.0 3.0 4.0 2 6.0 7.0 8.0 9.0 3 NaN NaN NaN NaN 4 NaN NaN NaN NaN >>> right A B C D 1 NaN NaN NaN NaN 2 10.0 20.0 30.0 40.0 3 60.0 70.0 80.0 90.0 4 600.0 700.0 800.0 900.0
Finally, the default axis=None will align on both index and columns:
>>> left, right = df.align(other, join="outer", axis=None) >>> left A B C D E 1 4.0 2.0 NaN 1.0 3.0 2 9.0 7.0 NaN 6.0 8.0 3 NaN NaN NaN NaN NaN 4 NaN NaN NaN NaN NaN >>> right A B C D E 1 NaN NaN NaN NaN NaN 2 10.0 20.0 30.0 40.0 NaN 3 60.0 70.0 80.0 90.0 NaN 4 600.0 700.0 800.0 900.0 NaN
- all(axis=0, bool_only=None, skipna=True, level=None, **kwargs)¶
Return whether all elements are True, potentially over an axis.
Returns True unless there at least one element within a series or along a Dataframe axis that is False or equivalent (e.g. zero or empty).
- Parameters
axis ({0 or ‘index’, 1 or ‘columns’, None}, default 0) – Indicate which axis or axes should be reduced.
0 / ‘index’ : reduce the index, return a Series whose index is the original column labels.
1 / ‘columns’ : reduce the columns, return a Series whose index is the original index.
None : reduce all axes, return a scalar.
bool_only (bool, default None) – Include only boolean columns. If None, will attempt to use everything, then use only boolean data. Not implemented for Series.
skipna (bool, default True) – Exclude NA/null values. If the entire row/column is NA and skipna is True, then the result will be True, as for an empty row/column. If skipna is False, then NA are treated as True, because these are not equal to zero.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a Series.
**kwargs (any, default None) – Additional keywords have no effect but might be accepted for compatibility with NumPy.
- Returns
If level is specified, then, DataFrame is returned; otherwise, Series is returned.
- Return type
See also
Series.all
Return True if all elements are True.
DataFrame.any
Return True if one (or more) elements are True.
Examples
Series
>>> pd.Series([True, True]).all() True >>> pd.Series([True, False]).all() False >>> pd.Series([], dtype="float64").all() True >>> pd.Series([np.nan]).all() True >>> pd.Series([np.nan]).all(skipna=False) True
DataFrames
Create a dataframe from a dictionary.
>>> df = pd.DataFrame({'col1': [True, True], 'col2': [True, False]}) >>> df col1 col2 0 True True 1 True False
Default behaviour checks if column-wise values all return True.
>>> df.all() col1 True col2 False dtype: bool
Specify
axis='columns'
to check if row-wise values all return True.>>> df.all(axis='columns') 0 True 1 False dtype: bool
Or
axis=None
for whether every value is True.>>> df.all(axis=None) False
- any(axis=0, bool_only=None, skipna=True, level=None, **kwargs)¶
Return whether any element is True, potentially over an axis.
Returns False unless there is at least one element within a series or along a Dataframe axis that is True or equivalent (e.g. non-zero or non-empty).
- Parameters
axis ({0 or ‘index’, 1 or ‘columns’, None}, default 0) – Indicate which axis or axes should be reduced.
0 / ‘index’ : reduce the index, return a Series whose index is the original column labels.
1 / ‘columns’ : reduce the columns, return a Series whose index is the original index.
None : reduce all axes, return a scalar.
bool_only (bool, default None) – Include only boolean columns. If None, will attempt to use everything, then use only boolean data. Not implemented for Series.
skipna (bool, default True) – Exclude NA/null values. If the entire row/column is NA and skipna is True, then the result will be False, as for an empty row/column. If skipna is False, then NA are treated as True, because these are not equal to zero.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a Series.
**kwargs (any, default None) – Additional keywords have no effect but might be accepted for compatibility with NumPy.
- Returns
If level is specified, then, DataFrame is returned; otherwise, Series is returned.
- Return type
See also
numpy.any
Numpy version of this method.
Series.any
Return whether any element is True.
Series.all
Return whether all elements are True.
DataFrame.any
Return whether any element is True over requested axis.
DataFrame.all
Return whether all elements are True over requested axis.
Examples
Series
For Series input, the output is a scalar indicating whether any element is True.
>>> pd.Series([False, False]).any() False >>> pd.Series([True, False]).any() True >>> pd.Series([], dtype="float64").any() False >>> pd.Series([np.nan]).any() False >>> pd.Series([np.nan]).any(skipna=False) True
DataFrame
Whether each column contains at least one True element (the default).
>>> df = pd.DataFrame({"A": [1, 2], "B": [0, 2], "C": [0, 0]}) >>> df A B C 0 1 0 0 1 2 2 0
>>> df.any() A True B True C False dtype: bool
Aggregating over the columns.
>>> df = pd.DataFrame({"A": [True, False], "B": [1, 2]}) >>> df A B 0 True 1 1 False 2
>>> df.any(axis='columns') 0 True 1 True dtype: bool
>>> df = pd.DataFrame({"A": [True, False], "B": [1, 0]}) >>> df A B 0 True 1 1 False 0
>>> df.any(axis='columns') 0 True 1 False dtype: bool
Aggregating over the entire DataFrame with
axis=None
.>>> df.any(axis=None) True
any for an empty DataFrame is an empty Series.
>>> pd.DataFrame([]).any() Series([], dtype: bool)
- append(other, ignore_index=False, verify_integrity=False, sort=False)[source]¶
Append rows of other to the end of caller, returning a new object.
Deprecated since version 1.4.0: Use
concat()
instead. For further details see Deprecated DataFrame.append and Series.appendColumns in other that are not in the caller are added as new columns.
- Parameters
other (DataFrame or Series/dict-like object, or list of these) – The data to append.
ignore_index (bool, default False) – If True, the resulting axis will be labeled 0, 1, …, n - 1.
verify_integrity (bool, default False) – If True, raise ValueError on creating index with duplicates.
sort (bool, default False) – Sort columns if the columns of self and other are not aligned.
Changed in version 1.0.0: Changed to not sort by default.
- Returns
A new DataFrame consisting of the rows of caller and the rows of other.
- Return type
See also
concat
General function to concatenate DataFrame or Series objects.
Notes
If a list of dict/series is passed and the keys are all contained in the DataFrame’s index, the order of the columns in the resulting DataFrame will be unchanged.
Iteratively appending rows to a DataFrame can be more computationally intensive than a single concatenate. A better solution is to append those rows to a list and then concatenate the list with the original DataFrame all at once.
Examples
>>> df = pd.DataFrame([[1, 2], [3, 4]], columns=list('AB'), index=['x', 'y']) >>> df A B x 1 2 y 3 4 >>> df2 = pd.DataFrame([[5, 6], [7, 8]], columns=list('AB'), index=['x', 'y']) >>> df.append(df2) A B x 1 2 y 3 4 x 5 6 y 7 8
With ignore_index set to True:
>>> df.append(df2, ignore_index=True) A B 0 1 2 1 3 4 2 5 6 3 7 8
The following, while not recommended methods for generating DataFrames, show two ways to generate a DataFrame from multiple data sources.
Less efficient:
>>> df = pd.DataFrame(columns=['A']) >>> for i in range(5): ... df = df.append({'A': i}, ignore_index=True) >>> df A 0 0 1 1 2 2 3 3 4 4
More efficient:
>>> pd.concat([pd.DataFrame([i], columns=['A']) for i in range(5)], ... ignore_index=True) A 0 0 1 1 2 2 3 3 4 4
- apply(func, axis=0, raw=False, result_type=None, args=(), **kwargs)[source]¶
Apply a function along an axis of the DataFrame.
Objects passed to the function are Series objects whose index is either the DataFrame’s index (
axis=0
) or the DataFrame’s columns (axis=1
). By default (result_type=None
), the final return type is inferred from the return type of the applied function. Otherwise, it depends on the result_type argument.- Parameters
func (function) – Function to apply to each column or row.
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – Axis along which the function is applied:
0 or ‘index’: apply function to each column.
1 or ‘columns’: apply function to each row.
raw (bool, default False) – Determines if row or column is passed as a Series or ndarray object:
False
: passes each row or column as a Series to the function.True
: the passed function will receive ndarray objects instead. If you are just applying a NumPy reduction function this will achieve much better performance.
result_type ({‘expand’, ‘reduce’, ‘broadcast’, None}, default None) – These only act when
axis=1
(columns):‘expand’ : list-like results will be turned into columns.
‘reduce’ : returns a Series if possible rather than expanding list-like results. This is the opposite of ‘expand’.
‘broadcast’ : results will be broadcast to the original shape of the DataFrame, the original index and columns will be retained.
The default behaviour (None) depends on the return value of the applied function: list-like results will be returned as a Series of those. However if the apply function returns a Series these are expanded to columns.
args (tuple) – Positional arguments to pass to func in addition to the array/series.
**kwargs – Additional keyword arguments to pass as keywords arguments to func.
- Returns
Result of applying
func
along the given axis of the DataFrame.- Return type
See also
DataFrame.applymap
For elementwise operations.
DataFrame.aggregate
Only perform aggregating type operations.
DataFrame.transform
Only perform transforming type operations.
Notes
Functions that mutate the passed object can produce unexpected behavior or errors and are not supported. See Mutating with User Defined Function (UDF) methods for more details.
Examples
>>> df = pd.DataFrame([[4, 9]] * 3, columns=['A', 'B']) >>> df A B 0 4 9 1 4 9 2 4 9
Using a numpy universal function (in this case the same as
np.sqrt(df)
):>>> df.apply(np.sqrt) A B 0 2.0 3.0 1 2.0 3.0 2 2.0 3.0
Using a reducing function on either axis
>>> df.apply(np.sum, axis=0) A 12 B 27 dtype: int64
>>> df.apply(np.sum, axis=1) 0 13 1 13 2 13 dtype: int64
Returning a list-like will result in a Series
>>> df.apply(lambda x: [1, 2], axis=1) 0 [1, 2] 1 [1, 2] 2 [1, 2] dtype: object
Passing
result_type='expand'
will expand list-like results to columns of a Dataframe>>> df.apply(lambda x: [1, 2], axis=1, result_type='expand') 0 1 0 1 2 1 1 2 2 1 2
Returning a Series inside the function is similar to passing
result_type='expand'
. The resulting column names will be the Series index.>>> df.apply(lambda x: pd.Series([1, 2], index=['foo', 'bar']), axis=1) foo bar 0 1 2 1 1 2 2 1 2
Passing
result_type='broadcast'
will ensure the same shape result, whether list-like or scalar is returned by the function, and broadcast it along the axis. The resulting column names will be the originals.>>> df.apply(lambda x: [1, 2], axis=1, result_type='broadcast') A B 0 1 2 1 1 2 2 1 2
- applymap(func, na_action=None, **kwargs)[source]¶
Apply a function to a Dataframe elementwise.
This method applies a function that accepts and returns a scalar to every element of a DataFrame.
- Parameters
func (callable) – Python function, returns a single value from a single value.
na_action ({None, ‘ignore’}, default None) – If ‘ignore’, propagate NaN values, without passing them to func.
New in version 1.2.
**kwargs – Additional keyword arguments to pass as keywords arguments to func.
New in version 1.3.0.
- Returns
Transformed DataFrame.
- Return type
See also
DataFrame.apply
Apply a function along input axis of DataFrame.
Examples
>>> df = pd.DataFrame([[1, 2.12], [3.356, 4.567]]) >>> df 0 1 0 1.000 2.120 1 3.356 4.567
>>> df.applymap(lambda x: len(str(x))) 0 1 0 3 4 1 5 5
Like Series.map, NA values can be ignored:
>>> df_copy = df.copy() >>> df_copy.iloc[0, 0] = pd.NA >>> df_copy.applymap(lambda x: len(str(x)), na_action='ignore') 0 1 0 <NA> 4 1 5 5
Note that a vectorized version of func often exists, which will be much faster. You could square each number elementwise.
>>> df.applymap(lambda x: x**2) 0 1 0 1.000000 4.494400 1 11.262736 20.857489
But it’s better to avoid applymap in that case.
>>> df ** 2 0 1 0 1.000000 4.494400 1 11.262736 20.857489
- asfreq(freq, method=None, how=None, normalize=False, fill_value=None)[source]¶
Convert time series to specified frequency.
Returns the original data conformed to a new index with the specified frequency.
If the index of this DataFrame is a
PeriodIndex
, the new index is the result of transforming the original index withPeriodIndex.asfreq
(so the original index will map one-to-one to the new index).Otherwise, the new index will be equivalent to
pd.date_range(start, end, freq=freq)
wherestart
andend
are, respectively, the first and last entries in the original index (seepandas.date_range()
). The values corresponding to any timesteps in the new index which were not present in the original index will be null (NaN
), unless a method for filling such unknowns is provided (see themethod
parameter below).The
resample()
method is more appropriate if an operation on each group of timesteps (such as an aggregate) is necessary to represent the data at the new frequency.- Parameters
freq (DateOffset or str) – Frequency DateOffset or string.
method ({‘backfill’/’bfill’, ‘pad’/’ffill’}, default None) – Method to use for filling holes in reindexed Series (note this does not fill NaNs that already were present):
‘pad’ / ‘ffill’: propagate last valid observation forward to next valid
‘backfill’ / ‘bfill’: use NEXT valid observation to fill.
how ({‘start’, ‘end’}, default end) – For PeriodIndex only (see PeriodIndex.asfreq).
normalize (bool, default False) – Whether to reset output index to midnight.
fill_value (scalar, optional) – Value to use for missing values, applied during upsampling (note this does not fill NaNs that already were present).
- Returns
DataFrame object reindexed to the specified frequency.
- Return type
See also
reindex
Conform DataFrame to new index with optional filling logic.
Notes
To learn more about the frequency strings, please see this link.
Examples
Start by creating a series with 4 one minute timestamps.
>>> index = pd.date_range('1/1/2000', periods=4, freq='T') >>> series = pd.Series([0.0, None, 2.0, 3.0], index=index) >>> df = pd.DataFrame({'s': series}) >>> df s 2000-01-01 00:00:00 0.0 2000-01-01 00:01:00 NaN 2000-01-01 00:02:00 2.0 2000-01-01 00:03:00 3.0
Upsample the series into 30 second bins.
>>> df.asfreq(freq='30S') s 2000-01-01 00:00:00 0.0 2000-01-01 00:00:30 NaN 2000-01-01 00:01:00 NaN 2000-01-01 00:01:30 NaN 2000-01-01 00:02:00 2.0 2000-01-01 00:02:30 NaN 2000-01-01 00:03:00 3.0
Upsample again, providing a
fill value
.>>> df.asfreq(freq='30S', fill_value=9.0) s 2000-01-01 00:00:00 0.0 2000-01-01 00:00:30 9.0 2000-01-01 00:01:00 NaN 2000-01-01 00:01:30 9.0 2000-01-01 00:02:00 2.0 2000-01-01 00:02:30 9.0 2000-01-01 00:03:00 3.0
Upsample again, providing a
method
.>>> df.asfreq(freq='30S', method='bfill') s 2000-01-01 00:00:00 0.0 2000-01-01 00:00:30 NaN 2000-01-01 00:01:00 NaN 2000-01-01 00:01:30 2.0 2000-01-01 00:02:00 2.0 2000-01-01 00:02:30 3.0 2000-01-01 00:03:00 3.0
- assign(**kwargs)[source]¶
Assign new columns to a DataFrame.
Returns a new object with all original columns in addition to new ones. Existing columns that are re-assigned will be overwritten.
- Parameters
**kwargs (dict of {str: callable or Series}) – The column names are keywords. If the values are callable, they are computed on the DataFrame and assigned to the new columns. The callable must not change input DataFrame (though pandas doesn’t check it). If the values are not callable, (e.g. a Series, scalar, or array), they are simply assigned.
- Returns
A new DataFrame with the new columns in addition to all the existing columns.
- Return type
Notes
Assigning multiple columns within the same
assign
is possible. Later items in ‘**kwargs’ may refer to newly created or modified columns in ‘df’; items are computed and assigned into ‘df’ in order.Examples
>>> df = pd.DataFrame({'temp_c': [17.0, 25.0]}, ... index=['Portland', 'Berkeley']) >>> df temp_c Portland 17.0 Berkeley 25.0
Where the value is a callable, evaluated on df:
>>> df.assign(temp_f=lambda x: x.temp_c * 9 / 5 + 32) temp_c temp_f Portland 17.0 62.6 Berkeley 25.0 77.0
Alternatively, the same behavior can be achieved by directly referencing an existing Series or sequence:
>>> df.assign(temp_f=df['temp_c'] * 9 / 5 + 32) temp_c temp_f Portland 17.0 62.6 Berkeley 25.0 77.0
You can create multiple columns within the same assign where one of the columns depends on another one defined within the same assign:
>>> df.assign(temp_f=lambda x: x['temp_c'] * 9 / 5 + 32, ... temp_k=lambda x: (x['temp_f'] + 459.67) * 5 / 9) temp_c temp_f temp_k Portland 17.0 62.6 290.15 Berkeley 25.0 77.0 298.15
- property axes: list[pandas.core.indexes.base.Index]¶
Return a list representing the axes of the DataFrame.
It has the row axis labels and column axis labels as the only members. They are returned in that order.
Examples
>>> df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]}) >>> df.axes [RangeIndex(start=0, stop=2, step=1), Index(['col1', 'col2'], dtype='object')]
- bfill(axis=None, inplace=False, limit=None, downcast=None)[source]¶
Synonym for
DataFrame.fillna()
withmethod='bfill'
.- Returns
Object with missing values filled or None if
inplace=True
.- Return type
Series/DataFrame or None
- boxplot(column=None, by=None, ax=None, fontsize=None, rot=0, grid=True, figsize=None, layout=None, return_type=None, backend=None, **kwargs)¶
Make a box plot from DataFrame columns.
Make a box-and-whisker plot from DataFrame columns, optionally grouped by some other columns. A box plot is a method for graphically depicting groups of numerical data through their quartiles. The box extends from the Q1 to Q3 quartile values of the data, with a line at the median (Q2). The whiskers extend from the edges of box to show the range of the data. By default, they extend no more than 1.5 * IQR (IQR = Q3 - Q1) from the edges of the box, ending at the farthest data point within that interval. Outliers are plotted as separate dots.
For further details see Wikipedia’s entry for boxplot.
- Parameters
column (str or list of str, optional) – Column name or list of names, or vector. Can be any valid input to
pandas.DataFrame.groupby()
.by (str or array-like, optional) – Column in the DataFrame to
pandas.DataFrame.groupby()
. One box-plot will be done per value of columns in by.ax (object of class matplotlib.axes.Axes, optional) – The matplotlib axes to be used by boxplot.
fontsize (float or str) – Tick label font size in points or as a string (e.g., large).
rot (int or float, default 0) – The rotation angle of labels (in degrees) with respect to the screen coordinate system.
grid (bool, default True) – Setting this to True will show the grid.
figsize (A tuple (width, height) in inches) – The size of the figure to create in matplotlib.
layout (tuple (rows, columns), optional) – For example, (3, 5) will display the subplots using 3 columns and 5 rows, starting from the top-left.
return_type ({‘axes’, ‘dict’, ‘both’} or None, default ‘axes’) – The kind of object to return. The default is
axes
.‘axes’ returns the matplotlib axes the boxplot is drawn on.
‘dict’ returns a dictionary whose values are the matplotlib Lines of the boxplot.
‘both’ returns a namedtuple with the axes and dict.
when grouping with
by
, a Series mapping columns toreturn_type
is returned.If
return_type
is None, a NumPy array of axes with the same shape aslayout
is returned.
backend (str, default None) – Backend to use instead of the backend specified in the option
plotting.backend
. For instance, ‘matplotlib’. Alternatively, to specify theplotting.backend
for the whole session, setpd.options.plotting.backend
.New in version 1.0.0.
**kwargs – All other plotting keyword arguments to be passed to
matplotlib.pyplot.boxplot()
.
- Returns
See Notes.
- Return type
result
See also
Series.plot.hist
Make a histogram.
matplotlib.pyplot.boxplot
Matplotlib equivalent plot.
Notes
The return type depends on the return_type parameter:
‘axes’ : object of class matplotlib.axes.Axes
‘dict’ : dict of matplotlib.lines.Line2D objects
‘both’ : a namedtuple with structure (ax, lines)
For data grouped with
by
, return a Series of the above or a numpy array:array
(forreturn_type = None
)
Use
return_type='dict'
when you want to tweak the appearance of the lines after plotting. In this case a dict containing the Lines making up the boxes, caps, fliers, medians, and whiskers is returned.Examples
Boxplots can be created for every column in the dataframe by
df.boxplot()
or indicating the columns to be used:Boxplots of variables distributions grouped by the values of a third variable can be created using the option
by
. For instance:A list of strings (i.e.
['X', 'Y']
) can be passed to boxplot in order to group the data by combination of the variables in the x-axis:The layout of boxplot can be adjusted giving a tuple to
layout
:Additional formatting can be done to the boxplot, like suppressing the grid (
grid=False
), rotating the labels in the x-axis (i.e.rot=45
) or changing the fontsize (i.e.fontsize=15
):The parameter
return_type
can be used to select the type of element returned by boxplot. Whenreturn_type='axes'
is selected, the matplotlib axes on which the boxplot is drawn are returned:>>> boxplot = df.boxplot(column=['Col1', 'Col2'], return_type='axes') >>> type(boxplot) <class 'matplotlib.axes._subplots.AxesSubplot'>
When grouping with
by
, a Series mapping columns toreturn_type
is returned:>>> boxplot = df.boxplot(column=['Col1', 'Col2'], by='X', ... return_type='axes') >>> type(boxplot) <class 'pandas.core.series.Series'>
If
return_type
is None, a NumPy array of axes with the same shape aslayout
is returned:>>> boxplot = df.boxplot(column=['Col1', 'Col2'], by='X', ... return_type=None) >>> type(boxplot) <class 'numpy.ndarray'>
- clip(lower=None, upper=None, axis=None, inplace=False, *args, **kwargs)[source]¶
Trim values at input threshold(s).
Assigns values outside boundary to boundary values. Thresholds can be singular values or array like, and in the latter case the clipping is performed element-wise in the specified axis.
- Parameters
lower (float or array-like, default None) – Minimum threshold value. All values below this threshold will be set to it. A missing threshold (e.g NA) will not clip the value.
upper (float or array-like, default None) – Maximum threshold value. All values above this threshold will be set to it. A missing threshold (e.g NA) will not clip the value.
axis (int or str axis name, optional) – Align object with lower and upper along the given axis.
inplace (bool, default False) – Whether to perform the operation in place on the data.
*args, **kwargs – Additional keywords have no effect but might be accepted for compatibility with numpy.
- Returns
Same type as calling object with the values outside the clip boundaries replaced or None if
inplace=True
.- Return type
See also
Series.clip
Trim values at input threshold in series.
DataFrame.clip
Trim values at input threshold in dataframe.
numpy.clip
Clip (limit) the values in an array.
Examples
>>> data = {'col_0': [9, -3, 0, -1, 5], 'col_1': [-2, -7, 6, 8, -5]} >>> df = pd.DataFrame(data) >>> df col_0 col_1 0 9 -2 1 -3 -7 2 0 6 3 -1 8 4 5 -5
Clips per column using lower and upper thresholds:
>>> df.clip(-4, 6) col_0 col_1 0 6 -2 1 -3 -4 2 0 6 3 -1 6 4 5 -4
Clips using specific lower and upper thresholds per column element:
>>> t = pd.Series([2, -4, -1, 6, 3]) >>> t 0 2 1 -4 2 -1 3 6 4 3 dtype: int64
>>> df.clip(t, t + 4, axis=0) col_0 col_1 0 6 2 1 -3 -4 2 0 3 3 6 8 4 5 3
Clips using specific lower threshold per column element, with missing values:
>>> t = pd.Series([2, -4, np.NaN, 6, 3]) >>> t 0 2.0 1 -4.0 2 NaN 3 6.0 4 3.0 dtype: float64
>>> df.clip(t, axis=0) col_0 col_1 0 9 2 1 -3 -4 2 0 6 3 6 8 4 5 3
- columns: Index¶
The column labels of the DataFrame.
- combine(other, func, fill_value=None, overwrite=True)[source]¶
Perform column-wise combine with another DataFrame.
Combines a DataFrame with other DataFrame using func to element-wise combine columns. The row and column indexes of the resulting DataFrame will be the union of the two.
- Parameters
other (DataFrame) – The DataFrame to merge column-wise.
func (function) – Function that takes two series as inputs and return a Series or a scalar. Used to merge the two dataframes column by columns.
fill_value (scalar value, default None) – The value to fill NaNs with prior to passing any column to the merge func.
overwrite (bool, default True) – If True, columns in self that do not exist in other will be overwritten with NaNs.
- Returns
Combination of the provided DataFrames.
- Return type
See also
DataFrame.combine_first
Combine two DataFrame objects and default to non-null values in frame calling the method.
Examples
Combine using a simple function that chooses the smaller column.
>>> df1 = pd.DataFrame({'A': [0, 0], 'B': [4, 4]}) >>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]}) >>> take_smaller = lambda s1, s2: s1 if s1.sum() < s2.sum() else s2 >>> df1.combine(df2, take_smaller) A B 0 0 3 1 0 3
Example using a true element-wise combine function.
>>> df1 = pd.DataFrame({'A': [5, 0], 'B': [2, 4]}) >>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]}) >>> df1.combine(df2, np.minimum) A B 0 1 2 1 0 3
Using fill_value fills Nones prior to passing the column to the merge function.
>>> df1 = pd.DataFrame({'A': [0, 0], 'B': [None, 4]}) >>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]}) >>> df1.combine(df2, take_smaller, fill_value=-5) A B 0 0 -5.0 1 0 4.0
However, if the same element in both dataframes is None, that None is preserved
>>> df1 = pd.DataFrame({'A': [0, 0], 'B': [None, 4]}) >>> df2 = pd.DataFrame({'A': [1, 1], 'B': [None, 3]}) >>> df1.combine(df2, take_smaller, fill_value=-5) A B 0 0 -5.0 1 0 3.0
Example that demonstrates the use of overwrite and behavior when the axis differ between the dataframes.
>>> df1 = pd.DataFrame({'A': [0, 0], 'B': [4, 4]}) >>> df2 = pd.DataFrame({'B': [3, 3], 'C': [-10, 1], }, index=[1, 2]) >>> df1.combine(df2, take_smaller) A B C 0 NaN NaN NaN 1 NaN 3.0 -10.0 2 NaN 3.0 1.0
>>> df1.combine(df2, take_smaller, overwrite=False) A B C 0 0.0 NaN NaN 1 0.0 3.0 -10.0 2 NaN 3.0 1.0
Demonstrating the preference of the passed in dataframe.
>>> df2 = pd.DataFrame({'B': [3, 3], 'C': [1, 1], }, index=[1, 2]) >>> df2.combine(df1, take_smaller) A B C 0 0.0 NaN NaN 1 0.0 3.0 NaN 2 NaN 3.0 NaN
>>> df2.combine(df1, take_smaller, overwrite=False) A B C 0 0.0 NaN NaN 1 0.0 3.0 1.0 2 NaN 3.0 1.0
- combine_first(other)[source]¶
Update null elements with value in the same location in other.
Combine two DataFrame objects by filling null values in one DataFrame with non-null values from other DataFrame. The row and column indexes of the resulting DataFrame will be the union of the two.
- Parameters
other (DataFrame) – Provided DataFrame to use to fill null values.
- Returns
The result of combining the provided DataFrame with the other object.
- Return type
See also
DataFrame.combine
Perform series-wise operation on two DataFrames using a given function.
Examples
>>> df1 = pd.DataFrame({'A': [None, 0], 'B': [None, 4]}) >>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]}) >>> df1.combine_first(df2) A B 0 1.0 3.0 1 0.0 4.0
Null values still persist if the location of that null value does not exist in other
>>> df1 = pd.DataFrame({'A': [None, 0], 'B': [4, None]}) >>> df2 = pd.DataFrame({'B': [3, 3], 'C': [1, 1]}, index=[1, 2]) >>> df1.combine_first(df2) A B C 0 NaN 4.0 NaN 1 0.0 3.0 1.0 2 NaN 3.0 1.0
- compare(other, align_axis=1, keep_shape=False, keep_equal=False)[source]¶
Compare to another DataFrame and show the differences.
New in version 1.1.0.
- Parameters
other (DataFrame) – Object to compare with.
align_axis ({0 or ‘index’, 1 or ‘columns’}, default 1) – Determine which axis to align the comparison on.
- 0, or ‘index’Resulting differences are stacked vertically
with rows drawn alternately from self and other.
- 1, or ‘columns’Resulting differences are aligned horizontally
with columns drawn alternately from self and other.
keep_shape (bool, default False) – If true, all rows and columns are kept. Otherwise, only the ones with different values are kept.
keep_equal (bool, default False) – If true, the result keeps values that are equal. Otherwise, equal values are shown as NaNs.
- Returns
DataFrame that shows the differences stacked side by side.
The resulting index will be a MultiIndex with ‘self’ and ‘other’ stacked alternately at the inner level.
- Return type
- Raises
ValueError – When the two DataFrames don’t have identical labels or shape.
See also
Series.compare
Compare with another Series and show differences.
DataFrame.equals
Test whether two objects contain the same elements.
Notes
Matching NaNs will not appear as a difference.
Can only compare identically-labeled (i.e. same shape, identical row and column labels) DataFrames
Examples
>>> df = pd.DataFrame( ... { ... "col1": ["a", "a", "b", "b", "a"], ... "col2": [1.0, 2.0, 3.0, np.nan, 5.0], ... "col3": [1.0, 2.0, 3.0, 4.0, 5.0] ... }, ... columns=["col1", "col2", "col3"], ... ) >>> df col1 col2 col3 0 a 1.0 1.0 1 a 2.0 2.0 2 b 3.0 3.0 3 b NaN 4.0 4 a 5.0 5.0
>>> df2 = df.copy() >>> df2.loc[0, 'col1'] = 'c' >>> df2.loc[2, 'col3'] = 4.0 >>> df2 col1 col2 col3 0 c 1.0 1.0 1 a 2.0 2.0 2 b 3.0 4.0 3 b NaN 4.0 4 a 5.0 5.0
Align the differences on columns
>>> df.compare(df2) col1 col3 self other self other 0 a c NaN NaN 2 NaN NaN 3.0 4.0
Stack the differences on rows
>>> df.compare(df2, align_axis=0) col1 col3 0 self a NaN other c NaN 2 self NaN 3.0 other NaN 4.0
Keep the equal values
>>> df.compare(df2, keep_equal=True) col1 col3 self other self other 0 a c 1.0 1.0 2 b b 3.0 4.0
Keep all original rows and columns
>>> df.compare(df2, keep_shape=True) col1 col2 col3 self other self other self other 0 a c NaN NaN NaN NaN 1 NaN NaN NaN NaN NaN NaN 2 NaN NaN NaN NaN 3.0 4.0 3 NaN NaN NaN NaN NaN NaN 4 NaN NaN NaN NaN NaN NaN
Keep all original rows and columns and also all original values
>>> df.compare(df2, keep_shape=True, keep_equal=True) col1 col2 col3 self other self other self other 0 a c 1.0 1.0 1.0 1.0 1 a a 2.0 2.0 2.0 2.0 2 b b 3.0 3.0 3.0 4.0 3 b b NaN NaN 4.0 4.0 4 a a 5.0 5.0 5.0 5.0
- corr(method='pearson', min_periods=1)[source]¶
Compute pairwise correlation of columns, excluding NA/null values.
- Parameters
method ({‘pearson’, ‘kendall’, ‘spearman’} or callable) – Method of correlation:
pearson : standard correlation coefficient
kendall : Kendall Tau correlation coefficient
spearman : Spearman rank correlation
- callable: callable with input two 1d ndarrays
and returning a float. Note that the returned matrix from corr will have 1 along the diagonals and will be symmetric regardless of the callable’s behavior.
min_periods (int, optional) – Minimum number of observations required per pair of columns to have a valid result. Currently only available for Pearson and Spearman correlation.
- Returns
Correlation matrix.
- Return type
See also
DataFrame.corrwith
Compute pairwise correlation with another DataFrame or Series.
Series.corr
Compute the correlation between two Series.
Examples
>>> def histogram_intersection(a, b): ... v = np.minimum(a, b).sum().round(decimals=1) ... return v >>> df = pd.DataFrame([(.2, .3), (.0, .6), (.6, .0), (.2, .1)], ... columns=['dogs', 'cats']) >>> df.corr(method=histogram_intersection) dogs cats dogs 1.0 0.3 cats 0.3 1.0
- corrwith(other, axis=0, drop=False, method='pearson')[source]¶
Compute pairwise correlation.
Pairwise correlation is computed between rows or columns of DataFrame with rows or columns of Series or DataFrame. DataFrames are first aligned along both axes before computing the correlations.
- Parameters
other (DataFrame, Series) – Object with which to compute correlations.
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – The axis to use. 0 or ‘index’ to compute column-wise, 1 or ‘columns’ for row-wise.
drop (bool, default False) – Drop missing indices from result.
method ({‘pearson’, ‘kendall’, ‘spearman’} or callable) – Method of correlation:
pearson : standard correlation coefficient
kendall : Kendall Tau correlation coefficient
spearman : Spearman rank correlation
- callable: callable with input two 1d ndarrays
and returning a float.
- Returns
Pairwise correlations.
- Return type
See also
DataFrame.corr
Compute pairwise correlation of columns.
- count(axis=0, level=None, numeric_only=False)[source]¶
Count non-NA cells for each column or row.
The values None, NaN, NaT, and optionally numpy.inf (depending on pandas.options.mode.use_inf_as_na) are considered NA.
- Parameters
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – If 0 or ‘index’ counts are generated for each column. If 1 or ‘columns’ counts are generated for each row.
level (int or str, optional) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DataFrame. A str specifies the level name.
numeric_only (bool, default False) – Include only float, int or boolean data.
- Returns
For each column/row the number of non-NA/null entries. If level is specified returns a DataFrame.
- Return type
See also
Series.count
Number of non-NA elements in a Series.
DataFrame.value_counts
Count unique combinations of columns.
DataFrame.shape
Number of DataFrame rows and columns (including NA elements).
DataFrame.isna
Boolean same-sized DataFrame showing places of NA elements.
Examples
Constructing DataFrame from a dictionary:
>>> df = pd.DataFrame({"Person": ... ["John", "Myla", "Lewis", "John", "Myla"], ... "Age": [24., np.nan, 21., 33, 26], ... "Single": [False, True, True, True, False]}) >>> df Person Age Single 0 John 24.0 False 1 Myla NaN True 2 Lewis 21.0 True 3 John 33.0 True 4 Myla 26.0 False
Notice the uncounted NA values:
>>> df.count() Person 5 Age 4 Single 5 dtype: int64
Counts for each row:
>>> df.count(axis='columns') 0 3 1 2 2 3 3 3 4 3 dtype: int64
- cov(min_periods=None, ddof=1)[source]¶
Compute pairwise covariance of columns, excluding NA/null values.
Compute the pairwise covariance among the series of a DataFrame. The returned data frame is the covariance matrix of the columns of the DataFrame.
Both NA and null values are automatically excluded from the calculation. (See the note below about bias from missing values.) A threshold can be set for the minimum number of observations for each value created. Comparisons with observations below this threshold will be returned as
NaN
.This method is generally used for the analysis of time series data to understand the relationship between different measures across time.
- Parameters
min_periods (int, optional) – Minimum number of observations required per pair of columns to have a valid result.
ddof (int, default 1) – Delta degrees of freedom. The divisor used in calculations is
N - ddof
, whereN
represents the number of elements.New in version 1.1.0.
- Returns
The covariance matrix of the series of the DataFrame.
- Return type
See also
Series.cov
Compute covariance with another Series.
core.window.ExponentialMovingWindow.cov
Exponential weighted sample covariance.
core.window.Expanding.cov
Expanding sample covariance.
core.window.Rolling.cov
Rolling sample covariance.
Notes
Returns the covariance matrix of the DataFrame’s time series. The covariance is normalized by N-ddof.
For DataFrames that have Series that are missing data (assuming that data is missing at random) the returned covariance matrix will be an unbiased estimate of the variance and covariance between the member Series.
However, for many applications this estimate may not be acceptable because the estimate covariance matrix is not guaranteed to be positive semi-definite. This could lead to estimate correlations having absolute values which are greater than one, and/or a non-invertible covariance matrix. See Estimation of covariance matrices for more details.
Examples
>>> df = pd.DataFrame([(1, 2), (0, 3), (2, 0), (1, 1)], ... columns=['dogs', 'cats']) >>> df.cov() dogs cats dogs 0.666667 -1.000000 cats -1.000000 1.666667
>>> np.random.seed(42) >>> df = pd.DataFrame(np.random.randn(1000, 5), ... columns=['a', 'b', 'c', 'd', 'e']) >>> df.cov() a b c d e a 0.998438 -0.020161 0.059277 -0.008943 0.014144 b -0.020161 1.059352 -0.008543 -0.024738 0.009826 c 0.059277 -0.008543 1.010670 -0.001486 -0.000271 d -0.008943 -0.024738 -0.001486 0.921297 -0.013692 e 0.014144 0.009826 -0.000271 -0.013692 0.977795
Minimum number of periods
This method also supports an optional
min_periods
keyword that specifies the required minimum number of non-NA observations for each column pair in order to have a valid result:>>> np.random.seed(42) >>> df = pd.DataFrame(np.random.randn(20, 3), ... columns=['a', 'b', 'c']) >>> df.loc[df.index[:5], 'a'] = np.nan >>> df.loc[df.index[5:10], 'b'] = np.nan >>> df.cov(min_periods=12) a b c a 0.316741 NaN -0.150812 b NaN 1.248003 0.191417 c -0.150812 0.191417 0.895202
- cummax(axis=None, skipna=True, *args, **kwargs)¶
Return cumulative maximum over a DataFrame or Series axis.
Returns a DataFrame or Series of the same size containing the cumulative maximum.
- Parameters
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – The index or the name of the axis. 0 is equivalent to None or ‘index’.
skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA.
*args, **kwargs – Additional keywords have no effect but might be accepted for compatibility with NumPy.
- Returns
Return cumulative maximum of Series or DataFrame.
- Return type
See also
core.window.Expanding.max
Similar functionality but ignores
NaN
values.DataFrame.max
Return the maximum over DataFrame axis.
DataFrame.cummax
Return cumulative maximum over DataFrame axis.
DataFrame.cummin
Return cumulative minimum over DataFrame axis.
DataFrame.cumsum
Return cumulative sum over DataFrame axis.
DataFrame.cumprod
Return cumulative product over DataFrame axis.
Examples
Series
>>> s = pd.Series([2, np.nan, 5, -1, 0]) >>> s 0 2.0 1 NaN 2 5.0 3 -1.0 4 0.0 dtype: float64
By default, NA values are ignored.
>>> s.cummax() 0 2.0 1 NaN 2 5.0 3 5.0 4 5.0 dtype: float64
To include NA values in the operation, use
skipna=False
>>> s.cummax(skipna=False) 0 2.0 1 NaN 2 NaN 3 NaN 4 NaN dtype: float64
DataFrame
>>> df = pd.DataFrame([[2.0, 1.0], ... [3.0, np.nan], ... [1.0, 0.0]], ... columns=list('AB')) >>> df A B 0 2.0 1.0 1 3.0 NaN 2 1.0 0.0
By default, iterates over rows and finds the maximum in each column. This is equivalent to
axis=None
oraxis='index'
.>>> df.cummax() A B 0 2.0 1.0 1 3.0 NaN 2 3.0 1.0
To iterate over columns and find the maximum in each row, use
axis=1
>>> df.cummax(axis=1) A B 0 2.0 2.0 1 3.0 NaN 2 1.0 1.0
- cummin(axis=None, skipna=True, *args, **kwargs)¶
Return cumulative minimum over a DataFrame or Series axis.
Returns a DataFrame or Series of the same size containing the cumulative minimum.
- Parameters
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – The index or the name of the axis. 0 is equivalent to None or ‘index’.
skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA.
*args, **kwargs – Additional keywords have no effect but might be accepted for compatibility with NumPy.
- Returns
Return cumulative minimum of Series or DataFrame.
- Return type
See also
core.window.Expanding.min
Similar functionality but ignores
NaN
values.DataFrame.min
Return the minimum over DataFrame axis.
DataFrame.cummax
Return cumulative maximum over DataFrame axis.
DataFrame.cummin
Return cumulative minimum over DataFrame axis.
DataFrame.cumsum
Return cumulative sum over DataFrame axis.
DataFrame.cumprod
Return cumulative product over DataFrame axis.
Examples
Series
>>> s = pd.Series([2, np.nan, 5, -1, 0]) >>> s 0 2.0 1 NaN 2 5.0 3 -1.0 4 0.0 dtype: float64
By default, NA values are ignored.
>>> s.cummin() 0 2.0 1 NaN 2 2.0 3 -1.0 4 -1.0 dtype: float64
To include NA values in the operation, use
skipna=False
>>> s.cummin(skipna=False) 0 2.0 1 NaN 2 NaN 3 NaN 4 NaN dtype: float64
DataFrame
>>> df = pd.DataFrame([[2.0, 1.0], ... [3.0, np.nan], ... [1.0, 0.0]], ... columns=list('AB')) >>> df A B 0 2.0 1.0 1 3.0 NaN 2 1.0 0.0
By default, iterates over rows and finds the minimum in each column. This is equivalent to
axis=None
oraxis='index'
.>>> df.cummin() A B 0 2.0 1.0 1 2.0 NaN 2 1.0 0.0
To iterate over columns and find the minimum in each row, use
axis=1
>>> df.cummin(axis=1) A B 0 2.0 1.0 1 3.0 NaN 2 1.0 0.0
- cumprod(axis=None, skipna=True, *args, **kwargs)¶
Return cumulative product over a DataFrame or Series axis.
Returns a DataFrame or Series of the same size containing the cumulative product.
- Parameters
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – The index or the name of the axis. 0 is equivalent to None or ‘index’.
skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA.
*args, **kwargs – Additional keywords have no effect but might be accepted for compatibility with NumPy.
- Returns
Return cumulative product of Series or DataFrame.
- Return type
See also
core.window.Expanding.prod
Similar functionality but ignores
NaN
values.DataFrame.prod
Return the product over DataFrame axis.
DataFrame.cummax
Return cumulative maximum over DataFrame axis.
DataFrame.cummin
Return cumulative minimum over DataFrame axis.
DataFrame.cumsum
Return cumulative sum over DataFrame axis.
DataFrame.cumprod
Return cumulative product over DataFrame axis.
Examples
Series
>>> s = pd.Series([2, np.nan, 5, -1, 0]) >>> s 0 2.0 1 NaN 2 5.0 3 -1.0 4 0.0 dtype: float64
By default, NA values are ignored.
>>> s.cumprod() 0 2.0 1 NaN 2 10.0 3 -10.0 4 -0.0 dtype: float64
To include NA values in the operation, use
skipna=False
>>> s.cumprod(skipna=False) 0 2.0 1 NaN 2 NaN 3 NaN 4 NaN dtype: float64
DataFrame
>>> df = pd.DataFrame([[2.0, 1.0], ... [3.0, np.nan], ... [1.0, 0.0]], ... columns=list('AB')) >>> df A B 0 2.0 1.0 1 3.0 NaN 2 1.0 0.0
By default, iterates over rows and finds the product in each column. This is equivalent to
axis=None
oraxis='index'
.>>> df.cumprod() A B 0 2.0 1.0 1 6.0 NaN 2 6.0 0.0
To iterate over columns and find the product in each row, use
axis=1
>>> df.cumprod(axis=1) A B 0 2.0 2.0 1 3.0 NaN 2 1.0 0.0
- cumsum(axis=None, skipna=True, *args, **kwargs)¶
Return cumulative sum over a DataFrame or Series axis.
Returns a DataFrame or Series of the same size containing the cumulative sum.
- Parameters
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – The index or the name of the axis. 0 is equivalent to None or ‘index’.
skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA.
*args, **kwargs – Additional keywords have no effect but might be accepted for compatibility with NumPy.
- Returns
Return cumulative sum of Series or DataFrame.
- Return type
See also
core.window.Expanding.sum
Similar functionality but ignores
NaN
values.DataFrame.sum
Return the sum over DataFrame axis.
DataFrame.cummax
Return cumulative maximum over DataFrame axis.
DataFrame.cummin
Return cumulative minimum over DataFrame axis.
DataFrame.cumsum
Return cumulative sum over DataFrame axis.
DataFrame.cumprod
Return cumulative product over DataFrame axis.
Examples
Series
>>> s = pd.Series([2, np.nan, 5, -1, 0]) >>> s 0 2.0 1 NaN 2 5.0 3 -1.0 4 0.0 dtype: float64
By default, NA values are ignored.
>>> s.cumsum() 0 2.0 1 NaN 2 7.0 3 6.0 4 6.0 dtype: float64
To include NA values in the operation, use
skipna=False
>>> s.cumsum(skipna=False) 0 2.0 1 NaN 2 NaN 3 NaN 4 NaN dtype: float64
DataFrame
>>> df = pd.DataFrame([[2.0, 1.0], ... [3.0, np.nan], ... [1.0, 0.0]], ... columns=list('AB')) >>> df A B 0 2.0 1.0 1 3.0 NaN 2 1.0 0.0
By default, iterates over rows and finds the sum in each column. This is equivalent to
axis=None
oraxis='index'
.>>> df.cumsum() A B 0 2.0 1.0 1 5.0 NaN 2 6.0 1.0
To iterate over columns and find the sum in each row, use
axis=1
>>> df.cumsum(axis=1) A B 0 2.0 3.0 1 3.0 NaN 2 1.0 1.0
- diff(periods=1, axis=0)[source]¶
First discrete difference of element.
Calculates the difference of a Dataframe element compared with another element in the Dataframe (default is element in previous row).
- Parameters
periods (int, default 1) – Periods to shift for calculating difference, accepts negative values.
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – Take difference over rows (0) or columns (1).
- Returns
First differences of the Series.
- Return type
Dataframe
See also
Dataframe.pct_change
Percent change over given number of periods.
Dataframe.shift
Shift index by desired number of periods with an optional time freq.
Series.diff
First discrete difference of object.
Notes
For boolean dtypes, this uses
operator.xor()
rather thanoperator.sub()
. The result is calculated according to current dtype in Dataframe, however dtype of the result is always float64.Examples
Difference with previous row
>>> df = pd.DataFrame({'a': [1, 2, 3, 4, 5, 6], ... 'b': [1, 1, 2, 3, 5, 8], ... 'c': [1, 4, 9, 16, 25, 36]}) >>> df a b c 0 1 1 1 1 2 1 4 2 3 2 9 3 4 3 16 4 5 5 25 5 6 8 36
>>> df.diff() a b c 0 NaN NaN NaN 1 1.0 0.0 3.0 2 1.0 1.0 5.0 3 1.0 1.0 7.0 4 1.0 2.0 9.0 5 1.0 3.0 11.0
Difference with previous column
>>> df.diff(axis=1) a b c 0 NaN 0 0 1 NaN -1 3 2 NaN -1 7 3 NaN -1 13 4 NaN 0 20 5 NaN 2 28
Difference with 3rd previous row
>>> df.diff(periods=3) a b c 0 NaN NaN NaN 1 NaN NaN NaN 2 NaN NaN NaN 3 3.0 2.0 15.0 4 3.0 4.0 21.0 5 3.0 6.0 27.0
Difference with following row
>>> df.diff(periods=-1) a b c 0 -1.0 0.0 -3.0 1 -1.0 -1.0 -5.0 2 -1.0 -1.0 -7.0 3 -1.0 -2.0 -9.0 4 -1.0 -3.0 -11.0 5 NaN NaN NaN
Overflow in input dtype
>>> df = pd.DataFrame({'a': [1, 0]}, dtype=np.uint8) >>> df.diff() a 0 NaN 1 255.0
- div(other, axis='columns', level=None, fill_value=None)¶
Get Floating division of dataframe and other, element-wise (binary operator truediv).
Equivalent to
dataframe / other
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, rtruediv.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- divide(other, axis='columns', level=None, fill_value=None)¶
Get Floating division of dataframe and other, element-wise (binary operator truediv).
Equivalent to
dataframe / other
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, rtruediv.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- dot(other: pandas.core.series.Series) pandas.core.series.Series [source]¶
- dot(other: DataFrame | Index | ArrayLike) pandas.core.frame.DataFrame
Compute the matrix multiplication between the DataFrame and other.
This method computes the matrix product between the DataFrame and the values of an other Series, DataFrame or a numpy array.
It can also be called using
self @ other
in Python >= 3.5.- Parameters
other (Series, DataFrame or array-like) – The other object to compute the matrix product with.
- Returns
If other is a Series, return the matrix product between self and other as a Series. If other is a DataFrame or a numpy.array, return the matrix product of self and other in a DataFrame of a np.array.
- Return type
See also
Series.dot
Similar method for Series.
Notes
The dimensions of DataFrame and other must be compatible in order to compute the matrix multiplication. In addition, the column names of DataFrame and the index of other must contain the same values, as they will be aligned prior to the multiplication.
The dot method for Series computes the inner product, instead of the matrix product here.
Examples
Here we multiply a DataFrame with a Series.
>>> df = pd.DataFrame([[0, 1, -2, -1], [1, 1, 1, 1]]) >>> s = pd.Series([1, 1, 2, 1]) >>> df.dot(s) 0 -4 1 5 dtype: int64
Here we multiply a DataFrame with another DataFrame.
>>> other = pd.DataFrame([[0, 1], [1, 2], [-1, -1], [2, 0]]) >>> df.dot(other) 0 1 0 1 4 1 2 2
Note that the dot method give the same result as @
>>> df @ other 0 1 0 1 4 1 2 2
The dot method works also if other is an np.array.
>>> arr = np.array([[0, 1], [1, 2], [-1, -1], [2, 0]]) >>> df.dot(arr) 0 1 0 1 4 1 2 2
Note how shuffling of the objects does not change the result.
>>> s2 = s.reindex([1, 0, 2, 3]) >>> df.dot(s2) 0 -4 1 5 dtype: int64
- drop(labels=None, axis=0, index=None, columns=None, level=None, inplace=False, errors='raise')[source]¶
Drop specified labels from rows or columns.
Remove rows or columns by specifying label names and corresponding axis, or by specifying directly index or column names. When using a multi-index, labels on different levels can be removed by specifying the level. See the user guide <advanced.shown_levels> for more information about the now unused levels.
- Parameters
labels (single label or list-like) – Index or column labels to drop. A tuple will be used as a single label and not treated as a list-like.
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – Whether to drop labels from the index (0 or ‘index’) or columns (1 or ‘columns’).
index (single label or list-like) – Alternative to specifying axis (
labels, axis=0
is equivalent toindex=labels
).columns (single label or list-like) – Alternative to specifying axis (
labels, axis=1
is equivalent tocolumns=labels
).level (int or level name, optional) – For MultiIndex, level from which the labels will be removed.
inplace (bool, default False) – If False, return a copy. Otherwise, do operation inplace and return None.
errors ({‘ignore’, ‘raise’}, default ‘raise’) – If ‘ignore’, suppress error and only existing labels are dropped.
- Returns
DataFrame without the removed index or column labels or None if
inplace=True
.- Return type
DataFrame or None
- Raises
KeyError – If any of the labels is not found in the selected axis.
See also
DataFrame.loc
Label-location based indexer for selection by label.
DataFrame.dropna
Return DataFrame with labels on given axis omitted where (all or any) data are missing.
DataFrame.drop_duplicates
Return DataFrame with duplicate rows removed, optionally only considering certain columns.
Series.drop
Return Series with specified index labels removed.
Examples
>>> df = pd.DataFrame(np.arange(12).reshape(3, 4), ... columns=['A', 'B', 'C', 'D']) >>> df A B C D 0 0 1 2 3 1 4 5 6 7 2 8 9 10 11
Drop columns
>>> df.drop(['B', 'C'], axis=1) A D 0 0 3 1 4 7 2 8 11
>>> df.drop(columns=['B', 'C']) A D 0 0 3 1 4 7 2 8 11
Drop a row by index
>>> df.drop([0, 1]) A B C D 2 8 9 10 11
Drop columns and/or rows of MultiIndex DataFrame
>>> midx = pd.MultiIndex(levels=[['lama', 'cow', 'falcon'], ... ['speed', 'weight', 'length']], ... codes=[[0, 0, 0, 1, 1, 1, 2, 2, 2], ... [0, 1, 2, 0, 1, 2, 0, 1, 2]]) >>> df = pd.DataFrame(index=midx, columns=['big', 'small'], ... data=[[45, 30], [200, 100], [1.5, 1], [30, 20], ... [250, 150], [1.5, 0.8], [320, 250], ... [1, 0.8], [0.3, 0.2]]) >>> df big small lama speed 45.0 30.0 weight 200.0 100.0 length 1.5 1.0 cow speed 30.0 20.0 weight 250.0 150.0 length 1.5 0.8 falcon speed 320.0 250.0 weight 1.0 0.8 length 0.3 0.2
Drop a specific index combination from the MultiIndex DataFrame, i.e., drop the combination
'falcon'
and'weight'
, which deletes only the corresponding row>>> df.drop(index=('falcon', 'weight')) big small lama speed 45.0 30.0 weight 200.0 100.0 length 1.5 1.0 cow speed 30.0 20.0 weight 250.0 150.0 length 1.5 0.8 falcon speed 320.0 250.0 length 0.3 0.2
>>> df.drop(index='cow', columns='small') big lama speed 45.0 weight 200.0 length 1.5 falcon speed 320.0 weight 1.0 length 0.3
>>> df.drop(index='length', level=1) big small lama speed 45.0 30.0 weight 200.0 100.0 cow speed 30.0 20.0 weight 250.0 150.0 falcon speed 320.0 250.0 weight 1.0 0.8
- drop_duplicates(subset=None, keep='first', inplace=False, ignore_index=False)[source]¶
Return DataFrame with duplicate rows removed.
Considering certain columns is optional. Indexes, including time indexes are ignored.
- Parameters
subset (column label or sequence of labels, optional) – Only consider certain columns for identifying duplicates, by default use all of the columns.
keep ({‘first’, ‘last’, False}, default ‘first’) – Determines which duplicates (if any) to keep. -
first
: Drop duplicates except for the first occurrence. -last
: Drop duplicates except for the last occurrence. - False : Drop all duplicates.inplace (bool, default False) – Whether to drop duplicates in place or to return a copy.
ignore_index (bool, default False) – If True, the resulting axis will be labeled 0, 1, …, n - 1.
New in version 1.0.0.
- Returns
DataFrame with duplicates removed or None if
inplace=True
.- Return type
DataFrame or None
See also
DataFrame.value_counts
Count unique combinations of columns.
Examples
Consider dataset containing ramen rating.
>>> df = pd.DataFrame({ ... 'brand': ['Yum Yum', 'Yum Yum', 'Indomie', 'Indomie', 'Indomie'], ... 'style': ['cup', 'cup', 'cup', 'pack', 'pack'], ... 'rating': [4, 4, 3.5, 15, 5] ... }) >>> df brand style rating 0 Yum Yum cup 4.0 1 Yum Yum cup 4.0 2 Indomie cup 3.5 3 Indomie pack 15.0 4 Indomie pack 5.0
By default, it removes duplicate rows based on all columns.
>>> df.drop_duplicates() brand style rating 0 Yum Yum cup 4.0 2 Indomie cup 3.5 3 Indomie pack 15.0 4 Indomie pack 5.0
To remove duplicates on specific column(s), use
subset
.>>> df.drop_duplicates(subset=['brand']) brand style rating 0 Yum Yum cup 4.0 2 Indomie cup 3.5
To remove duplicates and keep last occurrences, use
keep
.>>> df.drop_duplicates(subset=['brand', 'style'], keep='last') brand style rating 1 Yum Yum cup 4.0 2 Indomie cup 3.5 4 Indomie pack 5.0
- dropna(axis=0, how='any', thresh=None, subset=None, inplace=False)[source]¶
Remove missing values.
See the User Guide for more on which values are considered missing, and how to work with missing data.
- Parameters
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – Determine if rows or columns which contain missing values are removed.
0, or ‘index’ : Drop rows which contain missing values.
1, or ‘columns’ : Drop columns which contain missing value.
Changed in version 1.0.0: Pass tuple or list to drop on multiple axes. Only a single axis is allowed.
how ({‘any’, ‘all’}, default ‘any’) – Determine if row or column is removed from DataFrame, when we have at least one NA or all NA.
‘any’ : If any NA values are present, drop that row or column.
‘all’ : If all values are NA, drop that row or column.
thresh (int, optional) – Require that many non-NA values.
subset (column label or sequence of labels, optional) – Labels along other axis to consider, e.g. if you are dropping rows these would be a list of columns to include.
inplace (bool, default False) – If True, do operation inplace and return None.
- Returns
DataFrame with NA entries dropped from it or None if
inplace=True
.- Return type
DataFrame or None
See also
DataFrame.isna
Indicate missing values.
DataFrame.notna
Indicate existing (non-missing) values.
DataFrame.fillna
Replace missing values.
Series.dropna
Drop missing values.
Index.dropna
Drop missing indices.
Examples
>>> df = pd.DataFrame({"name": ['Alfred', 'Batman', 'Catwoman'], ... "toy": [np.nan, 'Batmobile', 'Bullwhip'], ... "born": [pd.NaT, pd.Timestamp("1940-04-25"), ... pd.NaT]}) >>> df name toy born 0 Alfred NaN NaT 1 Batman Batmobile 1940-04-25 2 Catwoman Bullwhip NaT
Drop the rows where at least one element is missing.
>>> df.dropna() name toy born 1 Batman Batmobile 1940-04-25
Drop the columns where at least one element is missing.
>>> df.dropna(axis='columns') name 0 Alfred 1 Batman 2 Catwoman
Drop the rows where all elements are missing.
>>> df.dropna(how='all') name toy born 0 Alfred NaN NaT 1 Batman Batmobile 1940-04-25 2 Catwoman Bullwhip NaT
Keep only the rows with at least 2 non-NA values.
>>> df.dropna(thresh=2) name toy born 1 Batman Batmobile 1940-04-25 2 Catwoman Bullwhip NaT
Define in which columns to look for missing values.
>>> df.dropna(subset=['name', 'toy']) name toy born 1 Batman Batmobile 1940-04-25 2 Catwoman Bullwhip NaT
Keep the DataFrame with valid entries in the same variable.
>>> df.dropna(inplace=True) >>> df name toy born 1 Batman Batmobile 1940-04-25
- duplicated(subset=None, keep='first')[source]¶
Return boolean Series denoting duplicate rows.
Considering certain columns is optional.
- Parameters
subset (column label or sequence of labels, optional) – Only consider certain columns for identifying duplicates, by default use all of the columns.
keep ({‘first’, ‘last’, False}, default ‘first’) – Determines which duplicates (if any) to mark.
first
: Mark duplicates asTrue
except for the first occurrence.last
: Mark duplicates asTrue
except for the last occurrence.False : Mark all duplicates as
True
.
- Returns
Boolean series for each duplicated rows.
- Return type
See also
Index.duplicated
Equivalent method on index.
Series.duplicated
Equivalent method on Series.
Series.drop_duplicates
Remove duplicate values from Series.
DataFrame.drop_duplicates
Remove duplicate values from DataFrame.
Examples
Consider dataset containing ramen rating.
>>> df = pd.DataFrame({ ... 'brand': ['Yum Yum', 'Yum Yum', 'Indomie', 'Indomie', 'Indomie'], ... 'style': ['cup', 'cup', 'cup', 'pack', 'pack'], ... 'rating': [4, 4, 3.5, 15, 5] ... }) >>> df brand style rating 0 Yum Yum cup 4.0 1 Yum Yum cup 4.0 2 Indomie cup 3.5 3 Indomie pack 15.0 4 Indomie pack 5.0
By default, for each set of duplicated values, the first occurrence is set on False and all others on True.
>>> df.duplicated() 0 False 1 True 2 False 3 False 4 False dtype: bool
By using ‘last’, the last occurrence of each set of duplicated values is set on False and all others on True.
>>> df.duplicated(keep='last') 0 True 1 False 2 False 3 False 4 False dtype: bool
By setting
keep
on False, all duplicates are True.>>> df.duplicated(keep=False) 0 True 1 True 2 False 3 False 4 False dtype: bool
To find duplicates on specific column(s), use
subset
.>>> df.duplicated(subset=['brand']) 0 False 1 True 2 False 3 True 4 True dtype: bool
- eq(other, axis='columns', level=None)¶
Get Equal to of dataframe and other, element-wise (binary operator eq).
Among flexible wrappers (eq, ne, le, lt, ge, gt) to comparison operators.
Equivalent to ==, !=, <=, <, >=, > with support to choose axis (rows or columns) and level for comparison.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}, default ‘columns’) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’).
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
Result of the comparison.
- Return type
DataFrame of bool
See also
DataFrame.eq
Compare DataFrames for equality elementwise.
DataFrame.ne
Compare DataFrames for inequality elementwise.
DataFrame.le
Compare DataFrames for less than inequality or equality elementwise.
DataFrame.lt
Compare DataFrames for strictly less than inequality elementwise.
DataFrame.ge
Compare DataFrames for greater than inequality or equality elementwise.
DataFrame.gt
Compare DataFrames for strictly greater than inequality elementwise.
Notes
Mismatched indices will be unioned together. NaN values are considered different (i.e. NaN != NaN).
Examples
>>> df = pd.DataFrame({'cost': [250, 150, 100], ... 'revenue': [100, 250, 300]}, ... index=['A', 'B', 'C']) >>> df cost revenue A 250 100 B 150 250 C 100 300
Comparison with a scalar, using either the operator or method:
>>> df == 100 cost revenue A False True B False False C True False
>>> df.eq(100) cost revenue A False True B False False C True False
When other is a
Series
, the columns of a DataFrame are aligned with the index of other and broadcast:>>> df != pd.Series([100, 250], index=["cost", "revenue"]) cost revenue A True True B True False C False True
Use the method to control the broadcast axis:
>>> df.ne(pd.Series([100, 300], index=["A", "D"]), axis='index') cost revenue A True False B True True C True True D True True
When comparing to an arbitrary sequence, the number of columns must match the number elements in other:
>>> df == [250, 100] cost revenue A True True B False False C False False
Use the method to control the axis:
>>> df.eq([250, 250, 100], axis='index') cost revenue A True False B False True C True False
Compare to a DataFrame of different shape.
>>> other = pd.DataFrame({'revenue': [300, 250, 100, 150]}, ... index=['A', 'B', 'C', 'D']) >>> other revenue A 300 B 250 C 100 D 150
>>> df.gt(other) cost revenue A False False B False False C False True D False False
Compare to a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'cost': [250, 150, 100, 150, 300, 220], ... 'revenue': [100, 250, 300, 200, 175, 225]}, ... index=[['Q1', 'Q1', 'Q1', 'Q2', 'Q2', 'Q2'], ... ['A', 'B', 'C', 'A', 'B', 'C']]) >>> df_multindex cost revenue Q1 A 250 100 B 150 250 C 100 300 Q2 A 150 200 B 300 175 C 220 225
>>> df.le(df_multindex, level=1) cost revenue Q1 A True True B True True C True True Q2 A False True B True False C True False
- eval(expr, inplace=False, **kwargs)[source]¶
Evaluate a string describing operations on DataFrame columns.
Operates on columns only, not specific rows or elements. This allows eval to run arbitrary code, which can make you vulnerable to code injection if you pass user input to this function.
- Parameters
expr (str) – The expression string to evaluate.
inplace (bool, default False) – If the expression contains an assignment, whether to perform the operation inplace and mutate the existing DataFrame. Otherwise, a new DataFrame is returned.
**kwargs – See the documentation for
eval()
for complete details on the keyword arguments accepted byquery()
.
- Returns
The result of the evaluation or None if
inplace=True
.- Return type
ndarray, scalar, pandas object, or None
See also
DataFrame.query
Evaluates a boolean expression to query the columns of a frame.
DataFrame.assign
Can evaluate an expression or function to create new values for a column.
eval
Evaluate a Python expression as a string using various backends.
Notes
For more details see the API documentation for
eval()
. For detailed examples see enhancing performance with eval.Examples
>>> df = pd.DataFrame({'A': range(1, 6), 'B': range(10, 0, -2)}) >>> df A B 0 1 10 1 2 8 2 3 6 3 4 4 4 5 2 >>> df.eval('A + B') 0 11 1 10 2 9 3 8 4 7 dtype: int64
Assignment is allowed though by default the original DataFrame is not modified.
>>> df.eval('C = A + B') A B C 0 1 10 11 1 2 8 10 2 3 6 9 3 4 4 8 4 5 2 7 >>> df A B 0 1 10 1 2 8 2 3 6 3 4 4 4 5 2
Use
inplace=True
to modify the original DataFrame.>>> df.eval('C = A + B', inplace=True) >>> df A B C 0 1 10 11 1 2 8 10 2 3 6 9 3 4 4 8 4 5 2 7
Multiple columns can be assigned to using multi-line expressions:
>>> df.eval( ... ''' ... C = A + B ... D = A - B ... ''' ... ) A B C D 0 1 10 11 -9 1 2 8 10 -6 2 3 6 9 -3 3 4 4 8 0 4 5 2 7 3
- explode(column, ignore_index=False)[source]¶
Transform each element of a list-like to a row, replicating index values.
New in version 0.25.0.
- Parameters
column (IndexLabel) – Column(s) to explode. For multiple columns, specify a non-empty list with each element be str or tuple, and all specified columns their list-like data on same row of the frame must have matching length.
New in version 1.3.0: Multi-column explode
ignore_index (bool, default False) – If True, the resulting index will be labeled 0, 1, …, n - 1.
New in version 1.1.0.
- Returns
Exploded lists to rows of the subset columns; index will be duplicated for these rows.
- Return type
- Raises
ValueError : –
If columns of the frame are not unique. * If specified columns to explode is empty list. * If specified columns to explode have not matching count of elements rowwise in the frame.
See also
DataFrame.unstack
Pivot a level of the (necessarily hierarchical) index labels.
DataFrame.melt
Unpivot a DataFrame from wide format to long format.
Series.explode
Explode a DataFrame from list-like columns to long format.
Notes
This routine will explode list-likes including lists, tuples, sets, Series, and np.ndarray. The result dtype of the subset rows will be object. Scalars will be returned unchanged, and empty list-likes will result in a np.nan for that row. In addition, the ordering of rows in the output will be non-deterministic when exploding sets.
Reference the user guide for more examples.
Examples
>>> df = pd.DataFrame({'A': [[0, 1, 2], 'foo', [], [3, 4]], ... 'B': 1, ... 'C': [['a', 'b', 'c'], np.nan, [], ['d', 'e']]}) >>> df A B C 0 [0, 1, 2] 1 [a, b, c] 1 foo 1 NaN 2 [] 1 [] 3 [3, 4] 1 [d, e]
Single-column explode.
>>> df.explode('A') A B C 0 0 1 [a, b, c] 0 1 1 [a, b, c] 0 2 1 [a, b, c] 1 foo 1 NaN 2 NaN 1 [] 3 3 1 [d, e] 3 4 1 [d, e]
Multi-column explode.
>>> df.explode(list('AC')) A B C 0 0 1 a 0 1 1 b 0 2 1 c 1 foo 1 NaN 2 NaN 1 NaN 3 3 1 d 3 4 1 e
- ffill(axis=None, inplace=False, limit=None, downcast=None)[source]¶
Synonym for
DataFrame.fillna()
withmethod='ffill'
.- Returns
Object with missing values filled or None if
inplace=True
.- Return type
Series/DataFrame or None
- fillna(value=None, method: FillnaOptions | None = None, axis: Axis | None = None, inplace: Literal[False] = False, limit=None, downcast=None) DataFrame [source]¶
- fillna(value, method: FillnaOptions | None, axis: Axis | None, inplace: Literal[True], limit=None, downcast=None) None
- fillna(*, inplace: Literal[True], limit=None, downcast=None) None
- fillna(value, *, inplace: Literal[True], limit=None, downcast=None) None
- fillna(*, method: FillnaOptions | None, inplace: Literal[True], limit=None, downcast=None) None
- fillna(*, axis: Axis | None, inplace: Literal[True], limit=None, downcast=None) None
- fillna(*, method: FillnaOptions | None, axis: Axis | None, inplace: Literal[True], limit=None, downcast=None) None
- fillna(value, *, axis: Axis | None, inplace: Literal[True], limit=None, downcast=None) None
- fillna(value, method: FillnaOptions | None, *, inplace: Literal[True], limit=None, downcast=None) None
- fillna(value=None, method: FillnaOptions | None = None, axis: Axis | None = None, inplace: bool = False, limit=None, downcast=None) DataFrame | None
Fill NA/NaN values using the specified method.
- Parameters
value (scalar, dict, Series, or DataFrame) – Value to use to fill holes (e.g. 0), alternately a dict/Series/DataFrame of values specifying which value to use for each index (for a Series) or column (for a DataFrame). Values not in the dict/Series/DataFrame will not be filled. This value cannot be a list.
method ({‘backfill’, ‘bfill’, ‘pad’, ‘ffill’, None}, default None) – Method to use for filling holes in reindexed Series pad / ffill: propagate last valid observation forward to next valid backfill / bfill: use next valid observation to fill gap.
axis ({0 or ‘index’, 1 or ‘columns’}) – Axis along which to fill missing values.
inplace (bool, default False) – If True, fill in-place. Note: this will modify any other views on this object (e.g., a no-copy slice for a column in a DataFrame).
limit (int, default None) – If method is specified, this is the maximum number of consecutive NaN values to forward/backward fill. In other words, if there is a gap with more than this number of consecutive NaNs, it will only be partially filled. If method is not specified, this is the maximum number of entries along the entire axis where NaNs will be filled. Must be greater than 0 if not None.
downcast (dict, default is None) – A dict of item->dtype of what to downcast if possible, or the string ‘infer’ which will try to downcast to an appropriate equal type (e.g. float64 to int64 if possible).
- Returns
Object with missing values filled or None if
inplace=True
.- Return type
DataFrame or None
See also
interpolate
Fill NaN values using interpolation.
reindex
Conform object to new index.
asfreq
Convert TimeSeries to specified frequency.
Examples
>>> df = pd.DataFrame([[np.nan, 2, np.nan, 0], ... [3, 4, np.nan, 1], ... [np.nan, np.nan, np.nan, np.nan], ... [np.nan, 3, np.nan, 4]], ... columns=list("ABCD")) >>> df A B C D 0 NaN 2.0 NaN 0.0 1 3.0 4.0 NaN 1.0 2 NaN NaN NaN NaN 3 NaN 3.0 NaN 4.0
Replace all NaN elements with 0s.
>>> df.fillna(0) A B C D 0 0.0 2.0 0.0 0.0 1 3.0 4.0 0.0 1.0 2 0.0 0.0 0.0 0.0 3 0.0 3.0 0.0 4.0
We can also propagate non-null values forward or backward.
>>> df.fillna(method="ffill") A B C D 0 NaN 2.0 NaN 0.0 1 3.0 4.0 NaN 1.0 2 3.0 4.0 NaN 1.0 3 3.0 3.0 NaN 4.0
Replace all NaN elements in column ‘A’, ‘B’, ‘C’, and ‘D’, with 0, 1, 2, and 3 respectively.
>>> values = {"A": 0, "B": 1, "C": 2, "D": 3} >>> df.fillna(value=values) A B C D 0 0.0 2.0 2.0 0.0 1 3.0 4.0 2.0 1.0 2 0.0 1.0 2.0 3.0 3 0.0 3.0 2.0 4.0
Only replace the first NaN element.
>>> df.fillna(value=values, limit=1) A B C D 0 0.0 2.0 2.0 0.0 1 3.0 4.0 NaN 1.0 2 NaN 1.0 NaN 3.0 3 NaN 3.0 NaN 4.0
When filling using a DataFrame, replacement happens along the same column names and same indices
>>> df2 = pd.DataFrame(np.zeros((4, 4)), columns=list("ABCE")) >>> df.fillna(df2) A B C D 0 0.0 2.0 0.0 0.0 1 3.0 4.0 0.0 1.0 2 0.0 0.0 0.0 NaN 3 0.0 3.0 0.0 4.0
Note that column D is not affected since it is not present in df2.
- floordiv(other, axis='columns', level=None, fill_value=None)¶
Get Integer division of dataframe and other, element-wise (binary operator floordiv).
Equivalent to
dataframe // other
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, rfloordiv.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- classmethod from_dict(data, orient='columns', dtype=None, columns=None)[source]¶
Construct DataFrame from dict of array-like or dicts.
Creates DataFrame object from dictionary by columns or by index allowing dtype specification.
- Parameters
data (dict) – Of the form {field : array-like} or {field : dict}.
orient ({‘columns’, ‘index’, ‘tight’}, default ‘columns’) – The “orientation” of the data. If the keys of the passed dict should be the columns of the resulting DataFrame, pass ‘columns’ (default). Otherwise if the keys should be rows, pass ‘index’. If ‘tight’, assume a dict with keys [‘index’, ‘columns’, ‘data’, ‘index_names’, ‘column_names’].
New in version 1.4.0: ‘tight’ as an allowed value for the
orient
argumentdtype (dtype, default None) – Data type to force, otherwise infer.
columns (list, default None) – Column labels to use when
orient='index'
. Raises a ValueError if used withorient='columns'
ororient='tight'
.
- Return type
See also
DataFrame.from_records
DataFrame from structured ndarray, sequence of tuples or dicts, or DataFrame.
DataFrame
DataFrame object creation using constructor.
DataFrame.to_dict
Convert the DataFrame to a dictionary.
Examples
By default the keys of the dict become the DataFrame columns:
>>> data = {'col_1': [3, 2, 1, 0], 'col_2': ['a', 'b', 'c', 'd']} >>> pd.DataFrame.from_dict(data) col_1 col_2 0 3 a 1 2 b 2 1 c 3 0 d
Specify
orient='index'
to create the DataFrame using dictionary keys as rows:>>> data = {'row_1': [3, 2, 1, 0], 'row_2': ['a', 'b', 'c', 'd']} >>> pd.DataFrame.from_dict(data, orient='index') 0 1 2 3 row_1 3 2 1 0 row_2 a b c d
When using the ‘index’ orientation, the column names can be specified manually:
>>> pd.DataFrame.from_dict(data, orient='index', ... columns=['A', 'B', 'C', 'D']) A B C D row_1 3 2 1 0 row_2 a b c d
Specify
orient='tight'
to create the DataFrame using a ‘tight’ format:>>> data = {'index': [('a', 'b'), ('a', 'c')], ... 'columns': [('x', 1), ('y', 2)], ... 'data': [[1, 3], [2, 4]], ... 'index_names': ['n1', 'n2'], ... 'column_names': ['z1', 'z2']} >>> pd.DataFrame.from_dict(data, orient='tight') z1 x y z2 1 2 n1 n2 a b 1 3 c 2 4
- classmethod from_records(data, index=None, exclude=None, columns=None, coerce_float=False, nrows=None)[source]¶
Convert structured or record ndarray to DataFrame.
Creates a DataFrame object from a structured ndarray, sequence of tuples or dicts, or DataFrame.
- Parameters
data (structured ndarray, sequence of tuples or dicts, or DataFrame) – Structured input data.
index (str, list of fields, array-like) – Field of array to use as the index, alternately a specific set of input labels to use.
exclude (sequence, default None) – Columns or fields to exclude.
columns (sequence, default None) – Column names to use. If the passed data do not have names associated with them, this argument provides names for the columns. Otherwise this argument indicates the order of the columns in the result (any names not found in the data will become all-NA columns).
coerce_float (bool, default False) – Attempt to convert values of non-string, non-numeric objects (like decimal.Decimal) to floating point, useful for SQL result sets.
nrows (int, default None) – Number of rows to read if data is an iterator.
- Return type
See also
DataFrame.from_dict
DataFrame from dict of array-like or dicts.
DataFrame
DataFrame object creation using constructor.
Examples
Data can be provided as a structured ndarray:
>>> data = np.array([(3, 'a'), (2, 'b'), (1, 'c'), (0, 'd')], ... dtype=[('col_1', 'i4'), ('col_2', 'U1')]) >>> pd.DataFrame.from_records(data) col_1 col_2 0 3 a 1 2 b 2 1 c 3 0 d
Data can be provided as a list of dicts:
>>> data = [{'col_1': 3, 'col_2': 'a'}, ... {'col_1': 2, 'col_2': 'b'}, ... {'col_1': 1, 'col_2': 'c'}, ... {'col_1': 0, 'col_2': 'd'}] >>> pd.DataFrame.from_records(data) col_1 col_2 0 3 a 1 2 b 2 1 c 3 0 d
Data can be provided as a list of tuples with corresponding columns:
>>> data = [(3, 'a'), (2, 'b'), (1, 'c'), (0, 'd')] >>> pd.DataFrame.from_records(data, columns=['col_1', 'col_2']) col_1 col_2 0 3 a 1 2 b 2 1 c 3 0 d
- ge(other, axis='columns', level=None)¶
Get Greater than or equal to of dataframe and other, element-wise (binary operator ge).
Among flexible wrappers (eq, ne, le, lt, ge, gt) to comparison operators.
Equivalent to ==, !=, <=, <, >=, > with support to choose axis (rows or columns) and level for comparison.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}, default ‘columns’) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’).
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
Result of the comparison.
- Return type
DataFrame of bool
See also
DataFrame.eq
Compare DataFrames for equality elementwise.
DataFrame.ne
Compare DataFrames for inequality elementwise.
DataFrame.le
Compare DataFrames for less than inequality or equality elementwise.
DataFrame.lt
Compare DataFrames for strictly less than inequality elementwise.
DataFrame.ge
Compare DataFrames for greater than inequality or equality elementwise.
DataFrame.gt
Compare DataFrames for strictly greater than inequality elementwise.
Notes
Mismatched indices will be unioned together. NaN values are considered different (i.e. NaN != NaN).
Examples
>>> df = pd.DataFrame({'cost': [250, 150, 100], ... 'revenue': [100, 250, 300]}, ... index=['A', 'B', 'C']) >>> df cost revenue A 250 100 B 150 250 C 100 300
Comparison with a scalar, using either the operator or method:
>>> df == 100 cost revenue A False True B False False C True False
>>> df.eq(100) cost revenue A False True B False False C True False
When other is a
Series
, the columns of a DataFrame are aligned with the index of other and broadcast:>>> df != pd.Series([100, 250], index=["cost", "revenue"]) cost revenue A True True B True False C False True
Use the method to control the broadcast axis:
>>> df.ne(pd.Series([100, 300], index=["A", "D"]), axis='index') cost revenue A True False B True True C True True D True True
When comparing to an arbitrary sequence, the number of columns must match the number elements in other:
>>> df == [250, 100] cost revenue A True True B False False C False False
Use the method to control the axis:
>>> df.eq([250, 250, 100], axis='index') cost revenue A True False B False True C True False
Compare to a DataFrame of different shape.
>>> other = pd.DataFrame({'revenue': [300, 250, 100, 150]}, ... index=['A', 'B', 'C', 'D']) >>> other revenue A 300 B 250 C 100 D 150
>>> df.gt(other) cost revenue A False False B False False C False True D False False
Compare to a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'cost': [250, 150, 100, 150, 300, 220], ... 'revenue': [100, 250, 300, 200, 175, 225]}, ... index=[['Q1', 'Q1', 'Q1', 'Q2', 'Q2', 'Q2'], ... ['A', 'B', 'C', 'A', 'B', 'C']]) >>> df_multindex cost revenue Q1 A 250 100 B 150 250 C 100 300 Q2 A 150 200 B 300 175 C 220 225
>>> df.le(df_multindex, level=1) cost revenue Q1 A True True B True True C True True Q2 A False True B True False C True False
- groupby(by=None, axis=0, level=None, as_index=True, sort=True, group_keys=True, squeeze=NoDefault.no_default, observed=False, dropna=True)[source]¶
Group DataFrame using a mapper or by a Series of columns.
A groupby operation involves some combination of splitting the object, applying a function, and combining the results. This can be used to group large amounts of data and compute operations on these groups.
- Parameters
by (mapping, function, label, or list of labels) – Used to determine the groups for the groupby. If
by
is a function, it’s called on each value of the object’s index. If a dict or Series is passed, the Series or dict VALUES will be used to determine the groups (the Series’ values are first aligned; see.align()
method). If a list or ndarray of length equal to the selected axis is passed (see the groupby user guide), the values are used as-is to determine the groups. A label or list of labels may be passed to group by the columns inself
. Notice that a tuple is interpreted as a (single) key.axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – Split along rows (0) or columns (1).
level (int, level name, or sequence of such, default None) – If the axis is a MultiIndex (hierarchical), group by a particular level or levels.
as_index (bool, default True) – For aggregated output, return object with group labels as the index. Only relevant for DataFrame input. as_index=False is effectively “SQL-style” grouped output.
sort (bool, default True) – Sort group keys. Get better performance by turning this off. Note this does not influence the order of observations within each group. Groupby preserves the order of rows within each group.
group_keys (bool, default True) – When calling apply, add group keys to index to identify pieces.
squeeze (bool, default False) – Reduce the dimensionality of the return type if possible, otherwise return a consistent type.
Deprecated since version 1.1.0.
observed (bool, default False) – This only applies if any of the groupers are Categoricals. If True: only show observed values for categorical groupers. If False: show all values for categorical groupers.
dropna (bool, default True) – If True, and if group keys contain NA values, NA values together with row/column will be dropped. If False, NA values will also be treated as the key in groups.
New in version 1.1.0.
- Returns
Returns a groupby object that contains information about the groups.
- Return type
DataFrameGroupBy
See also
resample
Convenience method for frequency conversion and resampling of time series.
Notes
See the user guide for more detailed usage and examples, including splitting an object into groups, iterating through groups, selecting a group, aggregation, and more.
Examples
>>> df = pd.DataFrame({'Animal': ['Falcon', 'Falcon', ... 'Parrot', 'Parrot'], ... 'Max Speed': [380., 370., 24., 26.]}) >>> df Animal Max Speed 0 Falcon 380.0 1 Falcon 370.0 2 Parrot 24.0 3 Parrot 26.0 >>> df.groupby(['Animal']).mean() Max Speed Animal Falcon 375.0 Parrot 25.0
Hierarchical Indexes
We can groupby different levels of a hierarchical index using the level parameter:
>>> arrays = [['Falcon', 'Falcon', 'Parrot', 'Parrot'], ... ['Captive', 'Wild', 'Captive', 'Wild']] >>> index = pd.MultiIndex.from_arrays(arrays, names=('Animal', 'Type')) >>> df = pd.DataFrame({'Max Speed': [390., 350., 30., 20.]}, ... index=index) >>> df Max Speed Animal Type Falcon Captive 390.0 Wild 350.0 Parrot Captive 30.0 Wild 20.0 >>> df.groupby(level=0).mean() Max Speed Animal Falcon 370.0 Parrot 25.0 >>> df.groupby(level="Type").mean() Max Speed Type Captive 210.0 Wild 185.0
We can also choose to include NA in group keys or not by setting dropna parameter, the default setting is True.
>>> l = [[1, 2, 3], [1, None, 4], [2, 1, 3], [1, 2, 2]] >>> df = pd.DataFrame(l, columns=["a", "b", "c"])
>>> df.groupby(by=["b"]).sum() a c b 1.0 2 3 2.0 2 5
>>> df.groupby(by=["b"], dropna=False).sum() a c b 1.0 2 3 2.0 2 5 NaN 1 4
>>> l = [["a", 12, 12], [None, 12.3, 33.], ["b", 12.3, 123], ["a", 1, 1]] >>> df = pd.DataFrame(l, columns=["a", "b", "c"])
>>> df.groupby(by="a").sum() b c a a 13.0 13.0 b 12.3 123.0
>>> df.groupby(by="a", dropna=False).sum() b c a a 13.0 13.0 b 12.3 123.0 NaN 12.3 33.0
- gt(other, axis='columns', level=None)¶
Get Greater than of dataframe and other, element-wise (binary operator gt).
Among flexible wrappers (eq, ne, le, lt, ge, gt) to comparison operators.
Equivalent to ==, !=, <=, <, >=, > with support to choose axis (rows or columns) and level for comparison.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}, default ‘columns’) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’).
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
Result of the comparison.
- Return type
DataFrame of bool
See also
DataFrame.eq
Compare DataFrames for equality elementwise.
DataFrame.ne
Compare DataFrames for inequality elementwise.
DataFrame.le
Compare DataFrames for less than inequality or equality elementwise.
DataFrame.lt
Compare DataFrames for strictly less than inequality elementwise.
DataFrame.ge
Compare DataFrames for greater than inequality or equality elementwise.
DataFrame.gt
Compare DataFrames for strictly greater than inequality elementwise.
Notes
Mismatched indices will be unioned together. NaN values are considered different (i.e. NaN != NaN).
Examples
>>> df = pd.DataFrame({'cost': [250, 150, 100], ... 'revenue': [100, 250, 300]}, ... index=['A', 'B', 'C']) >>> df cost revenue A 250 100 B 150 250 C 100 300
Comparison with a scalar, using either the operator or method:
>>> df == 100 cost revenue A False True B False False C True False
>>> df.eq(100) cost revenue A False True B False False C True False
When other is a
Series
, the columns of a DataFrame are aligned with the index of other and broadcast:>>> df != pd.Series([100, 250], index=["cost", "revenue"]) cost revenue A True True B True False C False True
Use the method to control the broadcast axis:
>>> df.ne(pd.Series([100, 300], index=["A", "D"]), axis='index') cost revenue A True False B True True C True True D True True
When comparing to an arbitrary sequence, the number of columns must match the number elements in other:
>>> df == [250, 100] cost revenue A True True B False False C False False
Use the method to control the axis:
>>> df.eq([250, 250, 100], axis='index') cost revenue A True False B False True C True False
Compare to a DataFrame of different shape.
>>> other = pd.DataFrame({'revenue': [300, 250, 100, 150]}, ... index=['A', 'B', 'C', 'D']) >>> other revenue A 300 B 250 C 100 D 150
>>> df.gt(other) cost revenue A False False B False False C False True D False False
Compare to a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'cost': [250, 150, 100, 150, 300, 220], ... 'revenue': [100, 250, 300, 200, 175, 225]}, ... index=[['Q1', 'Q1', 'Q1', 'Q2', 'Q2', 'Q2'], ... ['A', 'B', 'C', 'A', 'B', 'C']]) >>> df_multindex cost revenue Q1 A 250 100 B 150 250 C 100 300 Q2 A 150 200 B 300 175 C 220 225
>>> df.le(df_multindex, level=1) cost revenue Q1 A True True B True True C True True Q2 A False True B True False C True False
- hist(column=None, by=None, grid=True, xlabelsize=None, xrot=None, ylabelsize=None, yrot=None, ax=None, sharex=False, sharey=False, figsize=None, layout=None, bins=10, backend=None, legend=False, **kwargs)¶
Make a histogram of the DataFrame’s columns.
A `histogram`_ is a representation of the distribution of data. This function calls
matplotlib.pyplot.hist()
, on each series in the DataFrame, resulting in one histogram per column.- Parameters
data (DataFrame) – The pandas object holding the data.
column (str or sequence, optional) – If passed, will be used to limit data to a subset of columns.
by (object, optional) – If passed, then used to form histograms for separate groups.
grid (bool, default True) – Whether to show axis grid lines.
xlabelsize (int, default None) – If specified changes the x-axis label size.
xrot (float, default None) – Rotation of x axis labels. For example, a value of 90 displays the x labels rotated 90 degrees clockwise.
ylabelsize (int, default None) – If specified changes the y-axis label size.
yrot (float, default None) – Rotation of y axis labels. For example, a value of 90 displays the y labels rotated 90 degrees clockwise.
ax (Matplotlib axes object, default None) – The axes to plot the histogram on.
sharex (bool, default True if ax is None else False) – In case subplots=True, share x axis and set some x axis labels to invisible; defaults to True if ax is None otherwise False if an ax is passed in. Note that passing in both an ax and sharex=True will alter all x axis labels for all subplots in a figure.
sharey (bool, default False) – In case subplots=True, share y axis and set some y axis labels to invisible.
figsize (tuple, optional) – The size in inches of the figure to create. Uses the value in matplotlib.rcParams by default.
layout (tuple, optional) – Tuple of (rows, columns) for the layout of the histograms.
bins (int or sequence, default 10) – Number of histogram bins to be used. If an integer is given, bins + 1 bin edges are calculated and returned. If bins is a sequence, gives bin edges, including left edge of first bin and right edge of last bin. In this case, bins is returned unmodified.
backend (str, default None) – Backend to use instead of the backend specified in the option
plotting.backend
. For instance, ‘matplotlib’. Alternatively, to specify theplotting.backend
for the whole session, setpd.options.plotting.backend
.New in version 1.0.0.
legend (bool, default False) – Whether to show the legend.
New in version 1.1.0.
**kwargs – All other plotting keyword arguments to be passed to
matplotlib.pyplot.hist()
.
- Return type
matplotlib.AxesSubplot or numpy.ndarray of them
See also
matplotlib.pyplot.hist
Plot a histogram using matplotlib.
Examples
This example draws a histogram based on the length and width of some animals, displayed in three bins
- idxmax(axis=0, skipna=True)[source]¶
Return index of first occurrence of maximum over requested axis.
NA/null values are excluded.
- Parameters
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – The axis to use. 0 or ‘index’ for row-wise, 1 or ‘columns’ for column-wise.
skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA.
- Returns
Indexes of maxima along the specified axis.
- Return type
- Raises
If the row/column is empty
See also
Series.idxmax
Return index of the maximum element.
Notes
This method is the DataFrame version of
ndarray.argmax
.Examples
Consider a dataset containing food consumption in Argentina.
>>> df = pd.DataFrame({'consumption': [10.51, 103.11, 55.48], ... 'co2_emissions': [37.2, 19.66, 1712]}, ... index=['Pork', 'Wheat Products', 'Beef'])
>>> df consumption co2_emissions Pork 10.51 37.20 Wheat Products 103.11 19.66 Beef 55.48 1712.00
By default, it returns the index for the maximum value in each column.
>>> df.idxmax() consumption Wheat Products co2_emissions Beef dtype: object
To return the index for the maximum value in each row, use
axis="columns"
.>>> df.idxmax(axis="columns") Pork co2_emissions Wheat Products consumption Beef co2_emissions dtype: object
- idxmin(axis=0, skipna=True)[source]¶
Return index of first occurrence of minimum over requested axis.
NA/null values are excluded.
- Parameters
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – The axis to use. 0 or ‘index’ for row-wise, 1 or ‘columns’ for column-wise.
skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA.
- Returns
Indexes of minima along the specified axis.
- Return type
- Raises
If the row/column is empty
See also
Series.idxmin
Return index of the minimum element.
Notes
This method is the DataFrame version of
ndarray.argmin
.Examples
Consider a dataset containing food consumption in Argentina.
>>> df = pd.DataFrame({'consumption': [10.51, 103.11, 55.48], ... 'co2_emissions': [37.2, 19.66, 1712]}, ... index=['Pork', 'Wheat Products', 'Beef'])
>>> df consumption co2_emissions Pork 10.51 37.20 Wheat Products 103.11 19.66 Beef 55.48 1712.00
By default, it returns the index for the minimum value in each column.
>>> df.idxmin() consumption Pork co2_emissions Wheat Products dtype: object
To return the index for the minimum value in each row, use
axis="columns"
.>>> df.idxmin(axis="columns") Pork consumption Wheat Products co2_emissions Beef consumption dtype: object
- index: Index¶
The index (row labels) of the DataFrame.
- info(verbose=None, buf=None, max_cols=None, memory_usage=None, show_counts=None, null_counts=None)[source]¶
Print a concise summary of a DataFrame.
This method prints information about a DataFrame including the index dtype and columns, non-null values and memory usage.
- Parameters
data (DataFrame) – DataFrame to print information about.
verbose (bool, optional) – Whether to print the full summary. By default, the setting in
pandas.options.display.max_info_columns
is followed.buf (writable buffer, defaults to sys.stdout) – Where to send the output. By default, the output is printed to sys.stdout. Pass a writable buffer if you need to further process the output. max_cols : int, optional When to switch from the verbose to the truncated output. If the DataFrame has more than max_cols columns, the truncated output is used. By default, the setting in
pandas.options.display.max_info_columns
is used.memory_usage (bool, str, optional) – Specifies whether total memory usage of the DataFrame elements (including the index) should be displayed. By default, this follows the
pandas.options.display.memory_usage
setting.True always show memory usage. False never shows memory usage. A value of ‘deep’ is equivalent to “True with deep introspection”. Memory usage is shown in human-readable units (base-2 representation). Without deep introspection a memory estimation is made based in column dtype and number of rows assuming values consume the same memory amount for corresponding dtypes. With deep memory introspection, a real memory usage calculation is performed at the cost of computational resources.
show_counts (bool, optional) – Whether to show the non-null counts. By default, this is shown only if the DataFrame is smaller than
pandas.options.display.max_info_rows
andpandas.options.display.max_info_columns
. A value of True always shows the counts, and False never shows the counts.null_counts (bool, optional) –
Deprecated since version 1.2.0: Use show_counts instead.
- Returns
This method prints a summary of a DataFrame and returns None.
- Return type
None
See also
DataFrame.describe
Generate descriptive statistics of DataFrame columns.
DataFrame.memory_usage
Memory usage of DataFrame columns.
Examples
>>> int_values = [1, 2, 3, 4, 5] >>> text_values = ['alpha', 'beta', 'gamma', 'delta', 'epsilon'] >>> float_values = [0.0, 0.25, 0.5, 0.75, 1.0] >>> df = pd.DataFrame({"int_col": int_values, "text_col": text_values, ... "float_col": float_values}) >>> df int_col text_col float_col 0 1 alpha 0.00 1 2 beta 0.25 2 3 gamma 0.50 3 4 delta 0.75 4 5 epsilon 1.00
Prints information of all columns:
>>> df.info(verbose=True) <class 'pandas.core.frame.DataFrame'> RangeIndex: 5 entries, 0 to 4 Data columns (total 3 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 int_col 5 non-null int64 1 text_col 5 non-null object 2 float_col 5 non-null float64 dtypes: float64(1), int64(1), object(1) memory usage: 248.0+ bytes
Prints a summary of columns count and its dtypes but not per column information:
>>> df.info(verbose=False) <class 'pandas.core.frame.DataFrame'> RangeIndex: 5 entries, 0 to 4 Columns: 3 entries, int_col to float_col dtypes: float64(1), int64(1), object(1) memory usage: 248.0+ bytes
Pipe output of DataFrame.info to buffer instead of sys.stdout, get buffer content and writes to a text file:
>>> import io >>> buffer = io.StringIO() >>> df.info(buf=buffer) >>> s = buffer.getvalue() >>> with open("df_info.txt", "w", ... encoding="utf-8") as f: ... f.write(s) 260
The memory_usage parameter allows deep introspection mode, specially useful for big DataFrames and fine-tune memory optimization:
>>> random_strings_array = np.random.choice(['a', 'b', 'c'], 10 ** 6) >>> df = pd.DataFrame({ ... 'column_1': np.random.choice(['a', 'b', 'c'], 10 ** 6), ... 'column_2': np.random.choice(['a', 'b', 'c'], 10 ** 6), ... 'column_3': np.random.choice(['a', 'b', 'c'], 10 ** 6) ... }) >>> df.info() <class 'pandas.core.frame.DataFrame'> RangeIndex: 1000000 entries, 0 to 999999 Data columns (total 3 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 column_1 1000000 non-null object 1 column_2 1000000 non-null object 2 column_3 1000000 non-null object dtypes: object(3) memory usage: 22.9+ MB
>>> df.info(memory_usage='deep') <class 'pandas.core.frame.DataFrame'> RangeIndex: 1000000 entries, 0 to 999999 Data columns (total 3 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 column_1 1000000 non-null object 1 column_2 1000000 non-null object 2 column_3 1000000 non-null object dtypes: object(3) memory usage: 165.9 MB
- insert(loc, column, value, allow_duplicates=False)[source]¶
Insert column into DataFrame at specified location.
Raises a ValueError if column is already contained in the DataFrame, unless allow_duplicates is set to True.
- Parameters
loc (int) – Insertion index. Must verify 0 <= loc <= len(columns).
column (str, number, or hashable object) – Label of the inserted column.
value (Scalar, Series, or array-like)
allow_duplicates (bool, optional default False)
See also
Index.insert
Insert new item by index.
Examples
>>> df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]}) >>> df col1 col2 0 1 3 1 2 4 >>> df.insert(1, "newcol", [99, 99]) >>> df col1 newcol col2 0 1 99 3 1 2 99 4 >>> df.insert(0, "col1", [100, 100], allow_duplicates=True) >>> df col1 col1 newcol col2 0 100 1 99 3 1 100 2 99 4
Notice that pandas uses index alignment in case of value from type Series:
>>> df.insert(0, "col0", pd.Series([5, 6], index=[1, 2])) >>> df col0 col1 col1 newcol col2 0 NaN 100 1 99 3 1 5.0 100 2 99 4
- Return type
None
- interpolate(method='linear', axis=0, limit=None, inplace=False, limit_direction=None, limit_area=None, downcast=None, **kwargs)[source]¶
Fill NaN values using an interpolation method.
Please note that only
method='linear'
is supported for DataFrame/Series with a MultiIndex.- Parameters
method (str, default ‘linear’) – Interpolation technique to use. One of:
‘linear’: Ignore the index and treat the values as equally spaced. This is the only method supported on MultiIndexes.
‘time’: Works on daily and higher resolution data to interpolate given length of interval.
‘index’, ‘values’: use the actual numerical values of the index.
‘pad’: Fill in NaNs using existing values.
‘nearest’, ‘zero’, ‘slinear’, ‘quadratic’, ‘cubic’, ‘spline’, ‘barycentric’, ‘polynomial’: Passed to scipy.interpolate.interp1d. These methods use the numerical values of the index. Both ‘polynomial’ and ‘spline’ require that you also specify an order (int), e.g.
df.interpolate(method='polynomial', order=5)
.‘krogh’, ‘piecewise_polynomial’, ‘spline’, ‘pchip’, ‘akima’, ‘cubicspline’: Wrappers around the SciPy interpolation methods of similar names. See Notes.
‘from_derivatives’: Refers to scipy.interpolate.BPoly.from_derivatives which replaces ‘piecewise_polynomial’ interpolation method in scipy 0.18.
axis ({{0 or ‘index’, 1 or ‘columns’, None}}, default None) – Axis to interpolate along.
limit (int, optional) – Maximum number of consecutive NaNs to fill. Must be greater than 0.
inplace (bool, default False) – Update the data in place if possible.
limit_direction ({{‘forward’, ‘backward’, ‘both’}}, Optional) – Consecutive NaNs will be filled in this direction.
- If limit is specified:
If ‘method’ is ‘pad’ or ‘ffill’, ‘limit_direction’ must be ‘forward’.
If ‘method’ is ‘backfill’ or ‘bfill’, ‘limit_direction’ must be ‘backwards’.
- If ‘limit’ is not specified:
If ‘method’ is ‘backfill’ or ‘bfill’, the default is ‘backward’
else the default is ‘forward’
Changed in version 1.1.0: raises ValueError if limit_direction is ‘forward’ or ‘both’ and method is ‘backfill’ or ‘bfill’. raises ValueError if limit_direction is ‘backward’ or ‘both’ and method is ‘pad’ or ‘ffill’.
limit_area ({{None, ‘inside’, ‘outside’}}, default None) – If limit is specified, consecutive NaNs will be filled with this restriction.
None
: No fill restriction.‘inside’: Only fill NaNs surrounded by valid values (interpolate).
‘outside’: Only fill NaNs outside valid values (extrapolate).
downcast (optional, ‘infer’ or None, defaults to None) – Downcast dtypes if possible.
``**kwargs`` (optional) – Keyword arguments to pass on to the interpolating function.
- Returns
Returns the same object type as the caller, interpolated at some or all
NaN
values or None ifinplace=True
.- Return type
See also
fillna
Fill missing values using different methods.
scipy.interpolate.Akima1DInterpolator
Piecewise cubic polynomials (Akima interpolator).
scipy.interpolate.BPoly.from_derivatives
Piecewise polynomial in the Bernstein basis.
scipy.interpolate.interp1d
Interpolate a 1-D function.
scipy.interpolate.KroghInterpolator
Interpolate polynomial (Krogh interpolator).
scipy.interpolate.PchipInterpolator
PCHIP 1-d monotonic cubic interpolation.
scipy.interpolate.CubicSpline
Cubic spline data interpolator.
Notes
The ‘krogh’, ‘piecewise_polynomial’, ‘spline’, ‘pchip’ and ‘akima’ methods are wrappers around the respective SciPy implementations of similar names. These use the actual numerical values of the index. For more information on their behavior, see the SciPy documentation and SciPy tutorial.
Examples
Filling in
NaN
in aSeries
via linear interpolation.>>> s = pd.Series([0, 1, np.nan, 3]) >>> s 0 0.0 1 1.0 2 NaN 3 3.0 dtype: float64 >>> s.interpolate() 0 0.0 1 1.0 2 2.0 3 3.0 dtype: float64
Filling in
NaN
in a Series by padding, but filling at most two consecutiveNaN
at a time.>>> s = pd.Series([np.nan, "single_one", np.nan, ... "fill_two_more", np.nan, np.nan, np.nan, ... 4.71, np.nan]) >>> s 0 NaN 1 single_one 2 NaN 3 fill_two_more 4 NaN 5 NaN 6 NaN 7 4.71 8 NaN dtype: object >>> s.interpolate(method='pad', limit=2) 0 NaN 1 single_one 2 single_one 3 fill_two_more 4 fill_two_more 5 fill_two_more 6 NaN 7 4.71 8 4.71 dtype: object
Filling in
NaN
in a Series via polynomial interpolation or splines: Both ‘polynomial’ and ‘spline’ methods require that you also specify anorder
(int).>>> s = pd.Series([0, 2, np.nan, 8]) >>> s.interpolate(method='polynomial', order=2) 0 0.000000 1 2.000000 2 4.666667 3 8.000000 dtype: float64
Fill the DataFrame forward (that is, going down) along each column using linear interpolation.
Note how the last entry in column ‘a’ is interpolated differently, because there is no entry after it to use for interpolation. Note how the first entry in column ‘b’ remains
NaN
, because there is no entry before it to use for interpolation.>>> df = pd.DataFrame([(0.0, np.nan, -1.0, 1.0), ... (np.nan, 2.0, np.nan, np.nan), ... (2.0, 3.0, np.nan, 9.0), ... (np.nan, 4.0, -4.0, 16.0)], ... columns=list('abcd')) >>> df a b c d 0 0.0 NaN -1.0 1.0 1 NaN 2.0 NaN NaN 2 2.0 3.0 NaN 9.0 3 NaN 4.0 -4.0 16.0 >>> df.interpolate(method='linear', limit_direction='forward', axis=0) a b c d 0 0.0 NaN -1.0 1.0 1 1.0 2.0 -2.0 5.0 2 2.0 3.0 -3.0 9.0 3 2.0 4.0 -4.0 16.0
Using polynomial interpolation.
>>> df['d'].interpolate(method='polynomial', order=2) 0 1.0 1 4.0 2 9.0 3 16.0 Name: d, dtype: float64
- isin(values)[source]¶
Whether each element in the DataFrame is contained in values.
- Parameters
values (iterable, Series, DataFrame or dict) – The result will only be true at a location if all the labels match. If values is a Series, that’s the index. If values is a dict, the keys must be the column names, which must match. If values is a DataFrame, then both the index and column labels must match.
- Returns
DataFrame of booleans showing whether each element in the DataFrame is contained in values.
- Return type
See also
DataFrame.eq
Equality test for DataFrame.
Series.isin
Equivalent method on Series.
Series.str.contains
Test if pattern or regex is contained within a string of a Series or Index.
Examples
>>> df = pd.DataFrame({'num_legs': [2, 4], 'num_wings': [2, 0]}, ... index=['falcon', 'dog']) >>> df num_legs num_wings falcon 2 2 dog 4 0
When
values
is a list check whether every value in the DataFrame is present in the list (which animals have 0 or 2 legs or wings)>>> df.isin([0, 2]) num_legs num_wings falcon True True dog False True
To check if
values
is not in the DataFrame, use the~
operator:>>> ~df.isin([0, 2]) num_legs num_wings falcon False False dog True False
When
values
is a dict, we can pass values to check for each column separately:>>> df.isin({'num_wings': [0, 3]}) num_legs num_wings falcon False False dog False True
When
values
is a Series or DataFrame the index and column must match. Note that ‘falcon’ does not match based on the number of legs in other.>>> other = pd.DataFrame({'num_legs': [8, 3], 'num_wings': [0, 2]}, ... index=['spider', 'falcon']) >>> df.isin(other) num_legs num_wings falcon False True dog False False
- isna()[source]¶
Detect missing values.
Return a boolean same-sized object indicating if the values are NA. NA values, such as None or
numpy.NaN
, gets mapped to True values. Everything else gets mapped to False values. Characters such as empty strings''
ornumpy.inf
are not considered NA values (unless you setpandas.options.mode.use_inf_as_na = True
).- Returns
Mask of bool values for each element in DataFrame that indicates whether an element is an NA value.
- Return type
See also
DataFrame.isnull
Alias of isna.
DataFrame.notna
Boolean inverse of isna.
DataFrame.dropna
Omit axes labels with missing values.
isna
Top-level isna.
Examples
Show which entries in a DataFrame are NA.
>>> df = pd.DataFrame(dict(age=[5, 6, np.NaN], ... born=[pd.NaT, pd.Timestamp('1939-05-27'), ... pd.Timestamp('1940-04-25')], ... name=['Alfred', 'Batman', ''], ... toy=[None, 'Batmobile', 'Joker'])) >>> df age born name toy 0 5.0 NaT Alfred None 1 6.0 1939-05-27 Batman Batmobile 2 NaN 1940-04-25 Joker
>>> df.isna() age born name toy 0 False True False True 1 False False False False 2 True False False False
Show which entries in a Series are NA.
>>> ser = pd.Series([5, 6, np.NaN]) >>> ser 0 5.0 1 6.0 2 NaN dtype: float64
>>> ser.isna() 0 False 1 False 2 True dtype: bool
- isnull()[source]¶
DataFrame.isnull is an alias for DataFrame.isna.
Detect missing values.
Return a boolean same-sized object indicating if the values are NA. NA values, such as None or
numpy.NaN
, gets mapped to True values. Everything else gets mapped to False values. Characters such as empty strings''
ornumpy.inf
are not considered NA values (unless you setpandas.options.mode.use_inf_as_na = True
).- Returns
Mask of bool values for each element in DataFrame that indicates whether an element is an NA value.
- Return type
See also
DataFrame.isnull
Alias of isna.
DataFrame.notna
Boolean inverse of isna.
DataFrame.dropna
Omit axes labels with missing values.
isna
Top-level isna.
Examples
Show which entries in a DataFrame are NA.
>>> df = pd.DataFrame(dict(age=[5, 6, np.NaN], ... born=[pd.NaT, pd.Timestamp('1939-05-27'), ... pd.Timestamp('1940-04-25')], ... name=['Alfred', 'Batman', ''], ... toy=[None, 'Batmobile', 'Joker'])) >>> df age born name toy 0 5.0 NaT Alfred None 1 6.0 1939-05-27 Batman Batmobile 2 NaN 1940-04-25 Joker
>>> df.isna() age born name toy 0 False True False True 1 False False False False 2 True False False False
Show which entries in a Series are NA.
>>> ser = pd.Series([5, 6, np.NaN]) >>> ser 0 5.0 1 6.0 2 NaN dtype: float64
>>> ser.isna() 0 False 1 False 2 True dtype: bool
- items()[source]¶
Iterate over (column name, Series) pairs.
Iterates over the DataFrame columns, returning a tuple with the column name and the content as a Series.
- Yields
label (object) – The column names for the DataFrame being iterated over.
content (Series) – The column entries belonging to each label, as a Series.
See also
DataFrame.iterrows
Iterate over DataFrame rows as (index, Series) pairs.
DataFrame.itertuples
Iterate over DataFrame rows as namedtuples of the values.
Examples
>>> df = pd.DataFrame({'species': ['bear', 'bear', 'marsupial'], ... 'population': [1864, 22000, 80000]}, ... index=['panda', 'polar', 'koala']) >>> df species population panda bear 1864 polar bear 22000 koala marsupial 80000 >>> for label, content in df.items(): ... print(f'label: {label}') ... print(f'content: {content}', sep='\n') ... label: species content: panda bear polar bear koala marsupial Name: species, dtype: object label: population content: panda 1864 polar 22000 koala 80000 Name: population, dtype: int64
- iteritems()[source]¶
Iterate over (column name, Series) pairs.
Iterates over the DataFrame columns, returning a tuple with the column name and the content as a Series.
- Yields
label (object) – The column names for the DataFrame being iterated over.
content (Series) – The column entries belonging to each label, as a Series.
See also
DataFrame.iterrows
Iterate over DataFrame rows as (index, Series) pairs.
DataFrame.itertuples
Iterate over DataFrame rows as namedtuples of the values.
Examples
>>> df = pd.DataFrame({'species': ['bear', 'bear', 'marsupial'], ... 'population': [1864, 22000, 80000]}, ... index=['panda', 'polar', 'koala']) >>> df species population panda bear 1864 polar bear 22000 koala marsupial 80000 >>> for label, content in df.items(): ... print(f'label: {label}') ... print(f'content: {content}', sep='\n') ... label: species content: panda bear polar bear koala marsupial Name: species, dtype: object label: population content: panda 1864 polar 22000 koala 80000 Name: population, dtype: int64
- iterrows()[source]¶
Iterate over DataFrame rows as (index, Series) pairs.
- Yields
index (label or tuple of label) – The index of the row. A tuple for a MultiIndex.
data (Series) – The data of the row as a Series.
See also
DataFrame.itertuples
Iterate over DataFrame rows as namedtuples of the values.
DataFrame.items
Iterate over (column name, Series) pairs.
Notes
Because
iterrows
returns a Series for each row, it does not preserve dtypes across the rows (dtypes are preserved across columns for DataFrames). For example,>>> df = pd.DataFrame([[1, 1.5]], columns=['int', 'float']) >>> row = next(df.iterrows())[1] >>> row int 1.0 float 1.5 Name: 0, dtype: float64 >>> print(row['int'].dtype) float64 >>> print(df['int'].dtype) int64
To preserve dtypes while iterating over the rows, it is better to use
itertuples()
which returns namedtuples of the values and which is generally faster thaniterrows
.You should never modify something you are iterating over. This is not guaranteed to work in all cases. Depending on the data types, the iterator returns a copy and not a view, and writing to it will have no effect.
- itertuples(index=True, name='Pandas')[source]¶
Iterate over DataFrame rows as namedtuples.
- Parameters
index (bool, default True) – If True, return the index as the first element of the tuple.
name (str or None, default “Pandas”) – The name of the returned namedtuples or None to return regular tuples.
- Returns
An object to iterate over namedtuples for each row in the DataFrame with the first field possibly being the index and following fields being the column values.
- Return type
iterator
See also
DataFrame.iterrows
Iterate over DataFrame rows as (index, Series) pairs.
DataFrame.items
Iterate over (column name, Series) pairs.
Notes
The column names will be renamed to positional names if they are invalid Python identifiers, repeated, or start with an underscore. On python versions < 3.7 regular tuples are returned for DataFrames with a large number of columns (>254).
Examples
>>> df = pd.DataFrame({'num_legs': [4, 2], 'num_wings': [0, 2]}, ... index=['dog', 'hawk']) >>> df num_legs num_wings dog 4 0 hawk 2 2 >>> for row in df.itertuples(): ... print(row) ... Pandas(Index='dog', num_legs=4, num_wings=0) Pandas(Index='hawk', num_legs=2, num_wings=2)
By setting the index parameter to False we can remove the index as the first element of the tuple:
>>> for row in df.itertuples(index=False): ... print(row) ... Pandas(num_legs=4, num_wings=0) Pandas(num_legs=2, num_wings=2)
With the name parameter set we set a custom name for the yielded namedtuples:
>>> for row in df.itertuples(name='Animal'): ... print(row) ... Animal(Index='dog', num_legs=4, num_wings=0) Animal(Index='hawk', num_legs=2, num_wings=2)
- join(other, on=None, how='left', lsuffix='', rsuffix='', sort=False)[source]¶
Join columns of another DataFrame.
Join columns with other DataFrame either on index or on a key column. Efficiently join multiple DataFrame objects by index at once by passing a list.
- Parameters
other (DataFrame, Series, or list of DataFrame) – Index should be similar to one of the columns in this one. If a Series is passed, its name attribute must be set, and that will be used as the column name in the resulting joined DataFrame.
on (str, list of str, or array-like, optional) – Column or index level name(s) in the caller to join on the index in other, otherwise joins index-on-index. If multiple values given, the other DataFrame must have a MultiIndex. Can pass an array as the join key if it is not already contained in the calling DataFrame. Like an Excel VLOOKUP operation.
how ({‘left’, ‘right’, ‘outer’, ‘inner’}, default ‘left’) – How to handle the operation of the two objects.
left: use calling frame’s index (or column if on is specified)
right: use other’s index.
outer: form union of calling frame’s index (or column if on is specified) with other’s index, and sort it. lexicographically.
inner: form intersection of calling frame’s index (or column if on is specified) with other’s index, preserving the order of the calling’s one.
cross: creates the cartesian product from both frames, preserves the order of the left keys.
New in version 1.2.0.
lsuffix (str, default ‘’) – Suffix to use from left frame’s overlapping columns.
rsuffix (str, default ‘’) – Suffix to use from right frame’s overlapping columns.
sort (bool, default False) – Order result DataFrame lexicographically by the join key. If False, the order of the join key depends on the join type (how keyword).
- Returns
A dataframe containing columns from both the caller and other.
- Return type
See also
DataFrame.merge
For column(s)-on-column(s) operations.
Notes
Parameters on, lsuffix, and rsuffix are not supported when passing a list of DataFrame objects.
Support for specifying index levels as the on parameter was added in version 0.23.0.
Examples
>>> df = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3', 'K4', 'K5'], ... 'A': ['A0', 'A1', 'A2', 'A3', 'A4', 'A5']})
>>> df key A 0 K0 A0 1 K1 A1 2 K2 A2 3 K3 A3 4 K4 A4 5 K5 A5
>>> other = pd.DataFrame({'key': ['K0', 'K1', 'K2'], ... 'B': ['B0', 'B1', 'B2']})
>>> other key B 0 K0 B0 1 K1 B1 2 K2 B2
Join DataFrames using their indexes.
>>> df.join(other, lsuffix='_caller', rsuffix='_other') key_caller A key_other B 0 K0 A0 K0 B0 1 K1 A1 K1 B1 2 K2 A2 K2 B2 3 K3 A3 NaN NaN 4 K4 A4 NaN NaN 5 K5 A5 NaN NaN
If we want to join using the key columns, we need to set key to be the index in both df and other. The joined DataFrame will have key as its index.
>>> df.set_index('key').join(other.set_index('key')) A B key K0 A0 B0 K1 A1 B1 K2 A2 B2 K3 A3 NaN K4 A4 NaN K5 A5 NaN
Another option to join using the key columns is to use the on parameter. DataFrame.join always uses other’s index but we can use any column in df. This method preserves the original DataFrame’s index in the result.
>>> df.join(other.set_index('key'), on='key') key A B 0 K0 A0 B0 1 K1 A1 B1 2 K2 A2 B2 3 K3 A3 NaN 4 K4 A4 NaN 5 K5 A5 NaN
Using non-unique key values shows how they are matched.
>>> df = pd.DataFrame({'key': ['K0', 'K1', 'K1', 'K3', 'K0', 'K1'], ... 'A': ['A0', 'A1', 'A2', 'A3', 'A4', 'A5']})
>>> df key A 0 K0 A0 1 K1 A1 2 K1 A2 3 K3 A3 4 K0 A4 5 K1 A5
>>> df.join(other.set_index('key'), on='key') key A B 0 K0 A0 B0 1 K1 A1 B1 2 K1 A2 B1 3 K3 A3 NaN 4 K0 A4 B0 5 K1 A5 B1
- kurt(axis: Axis | None | lib.NoDefault = NoDefault.no_default, skipna=True, level=None, numeric_only=None, **kwargs)¶
Return unbiased kurtosis over requested axis.
Kurtosis obtained using Fisher’s definition of kurtosis (kurtosis of normal == 0.0). Normalized by N-1.
- Parameters
axis ({index (0), columns (1)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a Series.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
**kwargs – Additional keyword arguments to be passed to the function.
- Return type
- kurtosis(axis: Axis | None | lib.NoDefault = NoDefault.no_default, skipna=True, level=None, numeric_only=None, **kwargs)¶
Return unbiased kurtosis over requested axis.
Kurtosis obtained using Fisher’s definition of kurtosis (kurtosis of normal == 0.0). Normalized by N-1.
- Parameters
axis ({index (0), columns (1)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a Series.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
**kwargs – Additional keyword arguments to be passed to the function.
- Return type
- le(other, axis='columns', level=None)¶
Get Less than or equal to of dataframe and other, element-wise (binary operator le).
Among flexible wrappers (eq, ne, le, lt, ge, gt) to comparison operators.
Equivalent to ==, !=, <=, <, >=, > with support to choose axis (rows or columns) and level for comparison.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}, default ‘columns’) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’).
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
Result of the comparison.
- Return type
DataFrame of bool
See also
DataFrame.eq
Compare DataFrames for equality elementwise.
DataFrame.ne
Compare DataFrames for inequality elementwise.
DataFrame.le
Compare DataFrames for less than inequality or equality elementwise.
DataFrame.lt
Compare DataFrames for strictly less than inequality elementwise.
DataFrame.ge
Compare DataFrames for greater than inequality or equality elementwise.
DataFrame.gt
Compare DataFrames for strictly greater than inequality elementwise.
Notes
Mismatched indices will be unioned together. NaN values are considered different (i.e. NaN != NaN).
Examples
>>> df = pd.DataFrame({'cost': [250, 150, 100], ... 'revenue': [100, 250, 300]}, ... index=['A', 'B', 'C']) >>> df cost revenue A 250 100 B 150 250 C 100 300
Comparison with a scalar, using either the operator or method:
>>> df == 100 cost revenue A False True B False False C True False
>>> df.eq(100) cost revenue A False True B False False C True False
When other is a
Series
, the columns of a DataFrame are aligned with the index of other and broadcast:>>> df != pd.Series([100, 250], index=["cost", "revenue"]) cost revenue A True True B True False C False True
Use the method to control the broadcast axis:
>>> df.ne(pd.Series([100, 300], index=["A", "D"]), axis='index') cost revenue A True False B True True C True True D True True
When comparing to an arbitrary sequence, the number of columns must match the number elements in other:
>>> df == [250, 100] cost revenue A True True B False False C False False
Use the method to control the axis:
>>> df.eq([250, 250, 100], axis='index') cost revenue A True False B False True C True False
Compare to a DataFrame of different shape.
>>> other = pd.DataFrame({'revenue': [300, 250, 100, 150]}, ... index=['A', 'B', 'C', 'D']) >>> other revenue A 300 B 250 C 100 D 150
>>> df.gt(other) cost revenue A False False B False False C False True D False False
Compare to a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'cost': [250, 150, 100, 150, 300, 220], ... 'revenue': [100, 250, 300, 200, 175, 225]}, ... index=[['Q1', 'Q1', 'Q1', 'Q2', 'Q2', 'Q2'], ... ['A', 'B', 'C', 'A', 'B', 'C']]) >>> df_multindex cost revenue Q1 A 250 100 B 150 250 C 100 300 Q2 A 150 200 B 300 175 C 220 225
>>> df.le(df_multindex, level=1) cost revenue Q1 A True True B True True C True True Q2 A False True B True False C True False
- lookup(row_labels, col_labels)[source]¶
Label-based “fancy indexing” function for DataFrame. Given equal-length arrays of row and column labels, return an array of the values corresponding to each (row, col) pair.
Deprecated since version 1.2.0: DataFrame.lookup is deprecated, use DataFrame.melt and DataFrame.loc instead. For further details see Looking up values by index/column labels.
- Parameters
row_labels (sequence) – The row labels to use for lookup.
col_labels (sequence) – The column labels to use for lookup.
- Returns
The found values.
- Return type
- lt(other, axis='columns', level=None)¶
Get Less than of dataframe and other, element-wise (binary operator lt).
Among flexible wrappers (eq, ne, le, lt, ge, gt) to comparison operators.
Equivalent to ==, !=, <=, <, >=, > with support to choose axis (rows or columns) and level for comparison.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}, default ‘columns’) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’).
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
Result of the comparison.
- Return type
DataFrame of bool
See also
DataFrame.eq
Compare DataFrames for equality elementwise.
DataFrame.ne
Compare DataFrames for inequality elementwise.
DataFrame.le
Compare DataFrames for less than inequality or equality elementwise.
DataFrame.lt
Compare DataFrames for strictly less than inequality elementwise.
DataFrame.ge
Compare DataFrames for greater than inequality or equality elementwise.
DataFrame.gt
Compare DataFrames for strictly greater than inequality elementwise.
Notes
Mismatched indices will be unioned together. NaN values are considered different (i.e. NaN != NaN).
Examples
>>> df = pd.DataFrame({'cost': [250, 150, 100], ... 'revenue': [100, 250, 300]}, ... index=['A', 'B', 'C']) >>> df cost revenue A 250 100 B 150 250 C 100 300
Comparison with a scalar, using either the operator or method:
>>> df == 100 cost revenue A False True B False False C True False
>>> df.eq(100) cost revenue A False True B False False C True False
When other is a
Series
, the columns of a DataFrame are aligned with the index of other and broadcast:>>> df != pd.Series([100, 250], index=["cost", "revenue"]) cost revenue A True True B True False C False True
Use the method to control the broadcast axis:
>>> df.ne(pd.Series([100, 300], index=["A", "D"]), axis='index') cost revenue A True False B True True C True True D True True
When comparing to an arbitrary sequence, the number of columns must match the number elements in other:
>>> df == [250, 100] cost revenue A True True B False False C False False
Use the method to control the axis:
>>> df.eq([250, 250, 100], axis='index') cost revenue A True False B False True C True False
Compare to a DataFrame of different shape.
>>> other = pd.DataFrame({'revenue': [300, 250, 100, 150]}, ... index=['A', 'B', 'C', 'D']) >>> other revenue A 300 B 250 C 100 D 150
>>> df.gt(other) cost revenue A False False B False False C False True D False False
Compare to a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'cost': [250, 150, 100, 150, 300, 220], ... 'revenue': [100, 250, 300, 200, 175, 225]}, ... index=[['Q1', 'Q1', 'Q1', 'Q2', 'Q2', 'Q2'], ... ['A', 'B', 'C', 'A', 'B', 'C']]) >>> df_multindex cost revenue Q1 A 250 100 B 150 250 C 100 300 Q2 A 150 200 B 300 175 C 220 225
>>> df.le(df_multindex, level=1) cost revenue Q1 A True True B True True C True True Q2 A False True B True False C True False
- mad(axis=None, skipna=True, level=None)¶
Return the mean absolute deviation of the values over the requested axis.
- Parameters
axis ({index (0), columns (1)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a Series.
- Return type
- mask(cond, other=nan, inplace=False, axis=None, level=None, errors='raise', try_cast=NoDefault.no_default)[source]¶
Replace values where the condition is True.
- Parameters
cond (bool Series/DataFrame, array-like, or callable) – Where cond is False, keep the original value. Where True, replace with corresponding value from other. If cond is callable, it is computed on the Series/DataFrame and should return boolean Series/DataFrame or array. The callable must not change input Series/DataFrame (though pandas doesn’t check it).
other (scalar, Series/DataFrame, or callable) – Entries where cond is True are replaced with corresponding value from other. If other is callable, it is computed on the Series/DataFrame and should return scalar or Series/DataFrame. The callable must not change input Series/DataFrame (though pandas doesn’t check it).
inplace (bool, default False) – Whether to perform the operation in place on the data.
axis (int, default None) – Alignment axis if needed.
level (int, default None) – Alignment level if needed.
errors (str, {‘raise’, ‘ignore’}, default ‘raise’) – Note that currently this parameter won’t affect the results and will always coerce to a suitable dtype.
‘raise’ : allow exceptions to be raised.
‘ignore’ : suppress exceptions. On error return original object.
try_cast (bool, default None) – Try to cast the result back to the input type (if possible).
Deprecated since version 1.3.0: Manually cast back if necessary.
- Return type
Same type as caller or None if
inplace=True
.
See also
DataFrame.where()
Return an object of same shape as self.
Notes
The mask method is an application of the if-then idiom. For each element in the calling DataFrame, if
cond
isFalse
the element is used; otherwise the corresponding element from the DataFrameother
is used.The signature for
DataFrame.where()
differs fromnumpy.where()
. Roughlydf1.where(m, df2)
is equivalent tonp.where(m, df1, df2)
.For further details and examples see the
mask
documentation in indexing.Examples
>>> s = pd.Series(range(5)) >>> s.where(s > 0) 0 NaN 1 1.0 2 2.0 3 3.0 4 4.0 dtype: float64 >>> s.mask(s > 0) 0 0.0 1 NaN 2 NaN 3 NaN 4 NaN dtype: float64
>>> s.where(s > 1, 10) 0 10 1 10 2 2 3 3 4 4 dtype: int64 >>> s.mask(s > 1, 10) 0 0 1 1 2 10 3 10 4 10 dtype: int64
>>> df = pd.DataFrame(np.arange(10).reshape(-1, 2), columns=['A', 'B']) >>> df A B 0 0 1 1 2 3 2 4 5 3 6 7 4 8 9 >>> m = df % 3 == 0 >>> df.where(m, -df) A B 0 0 -1 1 -2 3 2 -4 -5 3 6 -7 4 -8 9 >>> df.where(m, -df) == np.where(m, df, -df) A B 0 True True 1 True True 2 True True 3 True True 4 True True >>> df.where(m, -df) == df.mask(~m, -df) A B 0 True True 1 True True 2 True True 3 True True 4 True True
- max(axis: int | None | lib.NoDefault = NoDefault.no_default, skipna=True, level=None, numeric_only=None, **kwargs)¶
Return the maximum of the values over the requested axis.
If you want the index of the maximum, use
idxmax
. This is the equivalent of thenumpy.ndarray
methodargmax
.- Parameters
axis ({index (0), columns (1)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a Series.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
**kwargs – Additional keyword arguments to be passed to the function.
- Return type
See also
Series.sum
Return the sum.
Series.min
Return the minimum.
Series.max
Return the maximum.
Series.idxmin
Return the index of the minimum.
Series.idxmax
Return the index of the maximum.
DataFrame.sum
Return the sum over the requested axis.
DataFrame.min
Return the minimum over the requested axis.
DataFrame.max
Return the maximum over the requested axis.
DataFrame.idxmin
Return the index of the minimum over the requested axis.
DataFrame.idxmax
Return the index of the maximum over the requested axis.
Examples
>>> idx = pd.MultiIndex.from_arrays([ ... ['warm', 'warm', 'cold', 'cold'], ... ['dog', 'falcon', 'fish', 'spider']], ... names=['blooded', 'animal']) >>> s = pd.Series([4, 2, 0, 8], name='legs', index=idx) >>> s blooded animal warm dog 4 falcon 2 cold fish 0 spider 8 Name: legs, dtype: int64
>>> s.max() 8
- mean(axis: int | None | lib.NoDefault = NoDefault.no_default, skipna=True, level=None, numeric_only=None, **kwargs)¶
Return the mean of the values over the requested axis.
- Parameters
axis ({index (0), columns (1)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a Series.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
**kwargs – Additional keyword arguments to be passed to the function.
- Return type
- median(axis: int | None | lib.NoDefault = NoDefault.no_default, skipna=True, level=None, numeric_only=None, **kwargs)¶
Return the median of the values over the requested axis.
- Parameters
axis ({index (0), columns (1)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a Series.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
**kwargs – Additional keyword arguments to be passed to the function.
- Return type
- melt(id_vars=None, value_vars=None, var_name=None, value_name='value', col_level=None, ignore_index=True)[source]¶
Unpivot a DataFrame from wide to long format, optionally leaving identifiers set.
This function is useful to massage a DataFrame into a format where one or more columns are identifier variables (id_vars), while all other columns, considered measured variables (value_vars), are “unpivoted” to the row axis, leaving just two non-identifier columns, ‘variable’ and ‘value’.
- Parameters
id_vars (tuple, list, or ndarray, optional) – Column(s) to use as identifier variables.
value_vars (tuple, list, or ndarray, optional) – Column(s) to unpivot. If not specified, uses all columns that are not set as id_vars.
var_name (scalar) – Name to use for the ‘variable’ column. If None it uses
frame.columns.name
or ‘variable’.value_name (scalar, default ‘value’) – Name to use for the ‘value’ column.
col_level (int or str, optional) – If columns are a MultiIndex then use this level to melt.
ignore_index (bool, default True) – If True, original index is ignored. If False, the original index is retained. Index labels will be repeated as necessary.
New in version 1.1.0.
- Returns
Unpivoted DataFrame.
- Return type
See also
melt
Identical method.
pivot_table
Create a spreadsheet-style pivot table as a DataFrame.
DataFrame.pivot
Return reshaped DataFrame organized by given index / column values.
DataFrame.explode
Explode a DataFrame from list-like columns to long format.
Notes
Reference the user guide for more examples.
Examples
>>> df = pd.DataFrame({'A': {0: 'a', 1: 'b', 2: 'c'}, ... 'B': {0: 1, 1: 3, 2: 5}, ... 'C': {0: 2, 1: 4, 2: 6}}) >>> df A B C 0 a 1 2 1 b 3 4 2 c 5 6
>>> df.melt(id_vars=['A'], value_vars=['B']) A variable value 0 a B 1 1 b B 3 2 c B 5
>>> df.melt(id_vars=['A'], value_vars=['B', 'C']) A variable value 0 a B 1 1 b B 3 2 c B 5 3 a C 2 4 b C 4 5 c C 6
The names of ‘variable’ and ‘value’ columns can be customized:
>>> df.melt(id_vars=['A'], value_vars=['B'], ... var_name='myVarname', value_name='myValname') A myVarname myValname 0 a B 1 1 b B 3 2 c B 5
Original index values can be kept around:
>>> df.melt(id_vars=['A'], value_vars=['B', 'C'], ignore_index=False) A variable value 0 a B 1 1 b B 3 2 c B 5 0 a C 2 1 b C 4 2 c C 6
If you have multi-index columns:
>>> df.columns = [list('ABC'), list('DEF')] >>> df A B C D E F 0 a 1 2 1 b 3 4 2 c 5 6
>>> df.melt(col_level=0, id_vars=['A'], value_vars=['B']) A variable value 0 a B 1 1 b B 3 2 c B 5
>>> df.melt(id_vars=[('A', 'D')], value_vars=[('B', 'E')]) (A, D) variable_0 variable_1 value 0 a B E 1 1 b B E 3 2 c B E 5
- memory_usage(index=True, deep=False)[source]¶
Return the memory usage of each column in bytes.
The memory usage can optionally include the contribution of the index and elements of object dtype.
This value is displayed in DataFrame.info by default. This can be suppressed by setting
pandas.options.display.memory_usage
to False.- Parameters
index (bool, default True) – Specifies whether to include the memory usage of the DataFrame’s index in returned Series. If
index=True
, the memory usage of the index is the first item in the output.deep (bool, default False) – If True, introspect the data deeply by interrogating object dtypes for system-level memory consumption, and include it in the returned values.
- Returns
A Series whose index is the original column names and whose values is the memory usage of each column in bytes.
- Return type
See also
numpy.ndarray.nbytes
Total bytes consumed by the elements of an ndarray.
Series.memory_usage
Bytes consumed by a Series.
Categorical
Memory-efficient array for string values with many repeated values.
DataFrame.info
Concise summary of a DataFrame.
Examples
>>> dtypes = ['int64', 'float64', 'complex128', 'object', 'bool'] >>> data = dict([(t, np.ones(shape=5000, dtype=int).astype(t)) ... for t in dtypes]) >>> df = pd.DataFrame(data) >>> df.head() int64 float64 complex128 object bool 0 1 1.0 1.0+0.0j 1 True 1 1 1.0 1.0+0.0j 1 True 2 1 1.0 1.0+0.0j 1 True 3 1 1.0 1.0+0.0j 1 True 4 1 1.0 1.0+0.0j 1 True
>>> df.memory_usage() Index 128 int64 40000 float64 40000 complex128 80000 object 40000 bool 5000 dtype: int64
>>> df.memory_usage(index=False) int64 40000 float64 40000 complex128 80000 object 40000 bool 5000 dtype: int64
The memory footprint of object dtype columns is ignored by default:
>>> df.memory_usage(deep=True) Index 128 int64 40000 float64 40000 complex128 80000 object 180000 bool 5000 dtype: int64
Use a Categorical for efficient storage of an object-dtype column with many repeated values.
>>> df['object'].astype('category').memory_usage(deep=True) 5244
- merge(right, how='inner', on=None, left_on=None, right_on=None, left_index=False, right_index=False, sort=False, suffixes=('_x', '_y'), copy=True, indicator=False, validate=None)[source]¶
Merge DataFrame or named Series objects with a database-style join.
A named Series object is treated as a DataFrame with a single named column.
The join is done on columns or indexes. If joining columns on columns, the DataFrame indexes will be ignored. Otherwise if joining indexes on indexes or indexes on a column or columns, the index will be passed on. When performing a cross merge, no column specifications to merge on are allowed.
Warning
If both key columns contain rows where the key is a null value, those rows will be matched against each other. This is different from usual SQL join behaviour and can lead to unexpected results.
- Parameters
right (DataFrame or named Series) – Object to merge with.
how ({‘left’, ‘right’, ‘outer’, ‘inner’, ‘cross’}, default ‘inner’) – Type of merge to be performed.
left: use only keys from left frame, similar to a SQL left outer join; preserve key order.
right: use only keys from right frame, similar to a SQL right outer join; preserve key order.
outer: use union of keys from both frames, similar to a SQL full outer join; sort keys lexicographically.
inner: use intersection of keys from both frames, similar to a SQL inner join; preserve the order of the left keys.
cross: creates the cartesian product from both frames, preserves the order of the left keys.
New in version 1.2.0.
on (label or list) – Column or index level names to join on. These must be found in both DataFrames. If on is None and not merging on indexes then this defaults to the intersection of the columns in both DataFrames.
left_on (label or list, or array-like) – Column or index level names to join on in the left DataFrame. Can also be an array or list of arrays of the length of the left DataFrame. These arrays are treated as if they are columns.
right_on (label or list, or array-like) – Column or index level names to join on in the right DataFrame. Can also be an array or list of arrays of the length of the right DataFrame. These arrays are treated as if they are columns.
left_index (bool, default False) – Use the index from the left DataFrame as the join key(s). If it is a MultiIndex, the number of keys in the other DataFrame (either the index or a number of columns) must match the number of levels.
right_index (bool, default False) – Use the index from the right DataFrame as the join key. Same caveats as left_index.
sort (bool, default False) – Sort the join keys lexicographically in the result DataFrame. If False, the order of the join keys depends on the join type (how keyword).
suffixes (list-like, default is (“_x”, “_y”)) – A length-2 sequence where each element is optionally a string indicating the suffix to add to overlapping column names in left and right respectively. Pass a value of None instead of a string to indicate that the column name from left or right should be left as-is, with no suffix. At least one of the values must not be None.
copy (bool, default True) – If False, avoid copy if possible.
indicator (bool or str, default False) – If True, adds a column to the output DataFrame called “_merge” with information on the source of each row. The column can be given a different name by providing a string argument. The column will have a Categorical type with the value of “left_only” for observations whose merge key only appears in the left DataFrame, “right_only” for observations whose merge key only appears in the right DataFrame, and “both” if the observation’s merge key is found in both DataFrames.
validate (str, optional) – If specified, checks if merge is of specified type.
“one_to_one” or “1:1”: check if merge keys are unique in both left and right datasets.
“one_to_many” or “1:m”: check if merge keys are unique in left dataset.
“many_to_one” or “m:1”: check if merge keys are unique in right dataset.
“many_to_many” or “m:m”: allowed, but does not result in checks.
- Returns
A DataFrame of the two merged objects.
- Return type
See also
merge_ordered
Merge with optional filling/interpolation.
merge_asof
Merge on nearest keys.
DataFrame.join
Similar method using indices.
Notes
Support for specifying index levels as the on, left_on, and right_on parameters was added in version 0.23.0 Support for merging named Series objects was added in version 0.24.0
Examples
>>> df1 = pd.DataFrame({'lkey': ['foo', 'bar', 'baz', 'foo'], ... 'value': [1, 2, 3, 5]}) >>> df2 = pd.DataFrame({'rkey': ['foo', 'bar', 'baz', 'foo'], ... 'value': [5, 6, 7, 8]}) >>> df1 lkey value 0 foo 1 1 bar 2 2 baz 3 3 foo 5 >>> df2 rkey value 0 foo 5 1 bar 6 2 baz 7 3 foo 8
Merge df1 and df2 on the lkey and rkey columns. The value columns have the default suffixes, _x and _y, appended.
>>> df1.merge(df2, left_on='lkey', right_on='rkey') lkey value_x rkey value_y 0 foo 1 foo 5 1 foo 1 foo 8 2 foo 5 foo 5 3 foo 5 foo 8 4 bar 2 bar 6 5 baz 3 baz 7
Merge DataFrames df1 and df2 with specified left and right suffixes appended to any overlapping columns.
>>> df1.merge(df2, left_on='lkey', right_on='rkey', ... suffixes=('_left', '_right')) lkey value_left rkey value_right 0 foo 1 foo 5 1 foo 1 foo 8 2 foo 5 foo 5 3 foo 5 foo 8 4 bar 2 bar 6 5 baz 3 baz 7
Merge DataFrames df1 and df2, but raise an exception if the DataFrames have any overlapping columns.
>>> df1.merge(df2, left_on='lkey', right_on='rkey', suffixes=(False, False)) Traceback (most recent call last): ... ValueError: columns overlap but no suffix specified: Index(['value'], dtype='object')
>>> df1 = pd.DataFrame({'a': ['foo', 'bar'], 'b': [1, 2]}) >>> df2 = pd.DataFrame({'a': ['foo', 'baz'], 'c': [3, 4]}) >>> df1 a b 0 foo 1 1 bar 2 >>> df2 a c 0 foo 3 1 baz 4
>>> df1.merge(df2, how='inner', on='a') a b c 0 foo 1 3
>>> df1.merge(df2, how='left', on='a') a b c 0 foo 1 3.0 1 bar 2 NaN
>>> df1 = pd.DataFrame({'left': ['foo', 'bar']}) >>> df2 = pd.DataFrame({'right': [7, 8]}) >>> df1 left 0 foo 1 bar >>> df2 right 0 7 1 8
>>> df1.merge(df2, how='cross') left right 0 foo 7 1 foo 8 2 bar 7 3 bar 8
- min(axis: int | None | lib.NoDefault = NoDefault.no_default, skipna=True, level=None, numeric_only=None, **kwargs)¶
Return the minimum of the values over the requested axis.
If you want the index of the minimum, use
idxmin
. This is the equivalent of thenumpy.ndarray
methodargmin
.- Parameters
axis ({index (0), columns (1)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a Series.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
**kwargs – Additional keyword arguments to be passed to the function.
- Return type
See also
Series.sum
Return the sum.
Series.min
Return the minimum.
Series.max
Return the maximum.
Series.idxmin
Return the index of the minimum.
Series.idxmax
Return the index of the maximum.
DataFrame.sum
Return the sum over the requested axis.
DataFrame.min
Return the minimum over the requested axis.
DataFrame.max
Return the maximum over the requested axis.
DataFrame.idxmin
Return the index of the minimum over the requested axis.
DataFrame.idxmax
Return the index of the maximum over the requested axis.
Examples
>>> idx = pd.MultiIndex.from_arrays([ ... ['warm', 'warm', 'cold', 'cold'], ... ['dog', 'falcon', 'fish', 'spider']], ... names=['blooded', 'animal']) >>> s = pd.Series([4, 2, 0, 8], name='legs', index=idx) >>> s blooded animal warm dog 4 falcon 2 cold fish 0 spider 8 Name: legs, dtype: int64
>>> s.min() 0
- mod(other, axis='columns', level=None, fill_value=None)¶
Get Modulo of dataframe and other, element-wise (binary operator mod).
Equivalent to
dataframe % other
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, rmod.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- mode(axis=0, numeric_only=False, dropna=True)[source]¶
Get the mode(s) of each element along the selected axis.
The mode of a set of values is the value that appears most often. It can be multiple values.
- Parameters
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – The axis to iterate over while searching for the mode:
0 or ‘index’ : get mode of each column
1 or ‘columns’ : get mode of each row.
numeric_only (bool, default False) – If True, only apply to numeric columns.
dropna (bool, default True) – Don’t consider counts of NaN/NaT.
- Returns
The modes of each column or row.
- Return type
See also
Series.mode
Return the highest frequency value in a Series.
Series.value_counts
Return the counts of values in a Series.
Examples
>>> df = pd.DataFrame([('bird', 2, 2), ... ('mammal', 4, np.nan), ... ('arthropod', 8, 0), ... ('bird', 2, np.nan)], ... index=('falcon', 'horse', 'spider', 'ostrich'), ... columns=('species', 'legs', 'wings')) >>> df species legs wings falcon bird 2 2.0 horse mammal 4 NaN spider arthropod 8 0.0 ostrich bird 2 NaN
By default, missing values are not considered, and the mode of wings are both 0 and 2. Because the resulting DataFrame has two rows, the second row of
species
andlegs
containsNaN
.>>> df.mode() species legs wings 0 bird 2.0 0.0 1 NaN NaN 2.0
Setting
dropna=False
NaN
values are considered and they can be the mode (like for wings).>>> df.mode(dropna=False) species legs wings 0 bird 2 NaN
Setting
numeric_only=True
, only the mode of numeric columns is computed, and columns of other types are ignored.>>> df.mode(numeric_only=True) legs wings 0 2.0 0.0 1 NaN 2.0
To compute the mode over columns and not rows, use the axis parameter:
>>> df.mode(axis='columns', numeric_only=True) 0 1 falcon 2.0 NaN horse 4.0 NaN spider 0.0 8.0 ostrich 2.0 NaN
- mul(other, axis='columns', level=None, fill_value=None)¶
Get Multiplication of dataframe and other, element-wise (binary operator mul).
Equivalent to
dataframe * other
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, rmul.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- multiply(other, axis='columns', level=None, fill_value=None)¶
Get Multiplication of dataframe and other, element-wise (binary operator mul).
Equivalent to
dataframe * other
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, rmul.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- ne(other, axis='columns', level=None)¶
Get Not equal to of dataframe and other, element-wise (binary operator ne).
Among flexible wrappers (eq, ne, le, lt, ge, gt) to comparison operators.
Equivalent to ==, !=, <=, <, >=, > with support to choose axis (rows or columns) and level for comparison.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}, default ‘columns’) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’).
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
Result of the comparison.
- Return type
DataFrame of bool
See also
DataFrame.eq
Compare DataFrames for equality elementwise.
DataFrame.ne
Compare DataFrames for inequality elementwise.
DataFrame.le
Compare DataFrames for less than inequality or equality elementwise.
DataFrame.lt
Compare DataFrames for strictly less than inequality elementwise.
DataFrame.ge
Compare DataFrames for greater than inequality or equality elementwise.
DataFrame.gt
Compare DataFrames for strictly greater than inequality elementwise.
Notes
Mismatched indices will be unioned together. NaN values are considered different (i.e. NaN != NaN).
Examples
>>> df = pd.DataFrame({'cost': [250, 150, 100], ... 'revenue': [100, 250, 300]}, ... index=['A', 'B', 'C']) >>> df cost revenue A 250 100 B 150 250 C 100 300
Comparison with a scalar, using either the operator or method:
>>> df == 100 cost revenue A False True B False False C True False
>>> df.eq(100) cost revenue A False True B False False C True False
When other is a
Series
, the columns of a DataFrame are aligned with the index of other and broadcast:>>> df != pd.Series([100, 250], index=["cost", "revenue"]) cost revenue A True True B True False C False True
Use the method to control the broadcast axis:
>>> df.ne(pd.Series([100, 300], index=["A", "D"]), axis='index') cost revenue A True False B True True C True True D True True
When comparing to an arbitrary sequence, the number of columns must match the number elements in other:
>>> df == [250, 100] cost revenue A True True B False False C False False
Use the method to control the axis:
>>> df.eq([250, 250, 100], axis='index') cost revenue A True False B False True C True False
Compare to a DataFrame of different shape.
>>> other = pd.DataFrame({'revenue': [300, 250, 100, 150]}, ... index=['A', 'B', 'C', 'D']) >>> other revenue A 300 B 250 C 100 D 150
>>> df.gt(other) cost revenue A False False B False False C False True D False False
Compare to a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'cost': [250, 150, 100, 150, 300, 220], ... 'revenue': [100, 250, 300, 200, 175, 225]}, ... index=[['Q1', 'Q1', 'Q1', 'Q2', 'Q2', 'Q2'], ... ['A', 'B', 'C', 'A', 'B', 'C']]) >>> df_multindex cost revenue Q1 A 250 100 B 150 250 C 100 300 Q2 A 150 200 B 300 175 C 220 225
>>> df.le(df_multindex, level=1) cost revenue Q1 A True True B True True C True True Q2 A False True B True False C True False
- nlargest(n, columns, keep='first')[source]¶
Return the first n rows ordered by columns in descending order.
Return the first n rows with the largest values in columns, in descending order. The columns that are not specified are returned as well, but not used for ordering.
This method is equivalent to
df.sort_values(columns, ascending=False).head(n)
, but more performant.- Parameters
n (int) – Number of rows to return.
columns (label or list of labels) – Column label(s) to order by.
keep ({‘first’, ‘last’, ‘all’}, default ‘first’) – Where there are duplicate values:
first
: prioritize the first occurrence(s)last
: prioritize the last occurrence(s)all
: do not drop any duplicates, even it means selecting more than n items.
- Returns
The first n rows ordered by the given columns in descending order.
- Return type
See also
DataFrame.nsmallest
Return the first n rows ordered by columns in ascending order.
DataFrame.sort_values
Sort DataFrame by the values.
DataFrame.head
Return the first n rows without re-ordering.
Notes
This function cannot be used with all column types. For example, when specifying columns with object or category dtypes,
TypeError
is raised.Examples
>>> df = pd.DataFrame({'population': [59000000, 65000000, 434000, ... 434000, 434000, 337000, 11300, ... 11300, 11300], ... 'GDP': [1937894, 2583560 , 12011, 4520, 12128, ... 17036, 182, 38, 311], ... 'alpha-2': ["IT", "FR", "MT", "MV", "BN", ... "IS", "NR", "TV", "AI"]}, ... index=["Italy", "France", "Malta", ... "Maldives", "Brunei", "Iceland", ... "Nauru", "Tuvalu", "Anguilla"]) >>> df population GDP alpha-2 Italy 59000000 1937894 IT France 65000000 2583560 FR Malta 434000 12011 MT Maldives 434000 4520 MV Brunei 434000 12128 BN Iceland 337000 17036 IS Nauru 11300 182 NR Tuvalu 11300 38 TV Anguilla 11300 311 AI
In the following example, we will use
nlargest
to select the three rows having the largest values in column “population”.>>> df.nlargest(3, 'population') population GDP alpha-2 France 65000000 2583560 FR Italy 59000000 1937894 IT Malta 434000 12011 MT
When using
keep='last'
, ties are resolved in reverse order:>>> df.nlargest(3, 'population', keep='last') population GDP alpha-2 France 65000000 2583560 FR Italy 59000000 1937894 IT Brunei 434000 12128 BN
When using
keep='all'
, all duplicate items are maintained:>>> df.nlargest(3, 'population', keep='all') population GDP alpha-2 France 65000000 2583560 FR Italy 59000000 1937894 IT Malta 434000 12011 MT Maldives 434000 4520 MV Brunei 434000 12128 BN
To order by the largest values in column “population” and then “GDP”, we can specify multiple columns like in the next example.
>>> df.nlargest(3, ['population', 'GDP']) population GDP alpha-2 France 65000000 2583560 FR Italy 59000000 1937894 IT Brunei 434000 12128 BN
- notna()[source]¶
Detect existing (non-missing) values.
Return a boolean same-sized object indicating if the values are not NA. Non-missing values get mapped to True. Characters such as empty strings
''
ornumpy.inf
are not considered NA values (unless you setpandas.options.mode.use_inf_as_na = True
). NA values, such as None ornumpy.NaN
, get mapped to False values.- Returns
Mask of bool values for each element in DataFrame that indicates whether an element is not an NA value.
- Return type
See also
DataFrame.notnull
Alias of notna.
DataFrame.isna
Boolean inverse of notna.
DataFrame.dropna
Omit axes labels with missing values.
notna
Top-level notna.
Examples
Show which entries in a DataFrame are not NA.
>>> df = pd.DataFrame(dict(age=[5, 6, np.NaN], ... born=[pd.NaT, pd.Timestamp('1939-05-27'), ... pd.Timestamp('1940-04-25')], ... name=['Alfred', 'Batman', ''], ... toy=[None, 'Batmobile', 'Joker'])) >>> df age born name toy 0 5.0 NaT Alfred None 1 6.0 1939-05-27 Batman Batmobile 2 NaN 1940-04-25 Joker
>>> df.notna() age born name toy 0 True False True False 1 True True True True 2 False True True True
Show which entries in a Series are not NA.
>>> ser = pd.Series([5, 6, np.NaN]) >>> ser 0 5.0 1 6.0 2 NaN dtype: float64
>>> ser.notna() 0 True 1 True 2 False dtype: bool
- notnull()[source]¶
DataFrame.notnull is an alias for DataFrame.notna.
Detect existing (non-missing) values.
Return a boolean same-sized object indicating if the values are not NA. Non-missing values get mapped to True. Characters such as empty strings
''
ornumpy.inf
are not considered NA values (unless you setpandas.options.mode.use_inf_as_na = True
). NA values, such as None ornumpy.NaN
, get mapped to False values.- Returns
Mask of bool values for each element in DataFrame that indicates whether an element is not an NA value.
- Return type
See also
DataFrame.notnull
Alias of notna.
DataFrame.isna
Boolean inverse of notna.
DataFrame.dropna
Omit axes labels with missing values.
notna
Top-level notna.
Examples
Show which entries in a DataFrame are not NA.
>>> df = pd.DataFrame(dict(age=[5, 6, np.NaN], ... born=[pd.NaT, pd.Timestamp('1939-05-27'), ... pd.Timestamp('1940-04-25')], ... name=['Alfred', 'Batman', ''], ... toy=[None, 'Batmobile', 'Joker'])) >>> df age born name toy 0 5.0 NaT Alfred None 1 6.0 1939-05-27 Batman Batmobile 2 NaN 1940-04-25 Joker
>>> df.notna() age born name toy 0 True False True False 1 True True True True 2 False True True True
Show which entries in a Series are not NA.
>>> ser = pd.Series([5, 6, np.NaN]) >>> ser 0 5.0 1 6.0 2 NaN dtype: float64
>>> ser.notna() 0 True 1 True 2 False dtype: bool
- nsmallest(n, columns, keep='first')[source]¶
Return the first n rows ordered by columns in ascending order.
Return the first n rows with the smallest values in columns, in ascending order. The columns that are not specified are returned as well, but not used for ordering.
This method is equivalent to
df.sort_values(columns, ascending=True).head(n)
, but more performant.- Parameters
n (int) – Number of items to retrieve.
columns (list or str) – Column name or names to order by.
keep ({‘first’, ‘last’, ‘all’}, default ‘first’) – Where there are duplicate values:
first
: take the first occurrence.last
: take the last occurrence.all
: do not drop any duplicates, even it means selecting more than n items.
- Return type
See also
DataFrame.nlargest
Return the first n rows ordered by columns in descending order.
DataFrame.sort_values
Sort DataFrame by the values.
DataFrame.head
Return the first n rows without re-ordering.
Examples
>>> df = pd.DataFrame({'population': [59000000, 65000000, 434000, ... 434000, 434000, 337000, 337000, ... 11300, 11300], ... 'GDP': [1937894, 2583560 , 12011, 4520, 12128, ... 17036, 182, 38, 311], ... 'alpha-2': ["IT", "FR", "MT", "MV", "BN", ... "IS", "NR", "TV", "AI"]}, ... index=["Italy", "France", "Malta", ... "Maldives", "Brunei", "Iceland", ... "Nauru", "Tuvalu", "Anguilla"]) >>> df population GDP alpha-2 Italy 59000000 1937894 IT France 65000000 2583560 FR Malta 434000 12011 MT Maldives 434000 4520 MV Brunei 434000 12128 BN Iceland 337000 17036 IS Nauru 337000 182 NR Tuvalu 11300 38 TV Anguilla 11300 311 AI
In the following example, we will use
nsmallest
to select the three rows having the smallest values in column “population”.>>> df.nsmallest(3, 'population') population GDP alpha-2 Tuvalu 11300 38 TV Anguilla 11300 311 AI Iceland 337000 17036 IS
When using
keep='last'
, ties are resolved in reverse order:>>> df.nsmallest(3, 'population', keep='last') population GDP alpha-2 Anguilla 11300 311 AI Tuvalu 11300 38 TV Nauru 337000 182 NR
When using
keep='all'
, all duplicate items are maintained:>>> df.nsmallest(3, 'population', keep='all') population GDP alpha-2 Tuvalu 11300 38 TV Anguilla 11300 311 AI Iceland 337000 17036 IS Nauru 337000 182 NR
To order by the smallest values in column “population” and then “GDP”, we can specify multiple columns like in the next example.
>>> df.nsmallest(3, ['population', 'GDP']) population GDP alpha-2 Tuvalu 11300 38 TV Anguilla 11300 311 AI Nauru 337000 182 NR
- nunique(axis=0, dropna=True)[source]¶
Count number of distinct elements in specified axis.
Return Series with number of distinct elements. Can ignore NaN values.
- Parameters
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – The axis to use. 0 or ‘index’ for row-wise, 1 or ‘columns’ for column-wise.
dropna (bool, default True) – Don’t include NaN in the counts.
- Return type
See also
Series.nunique
Method nunique for Series.
DataFrame.count
Count non-NA cells for each column or row.
Examples
>>> df = pd.DataFrame({'A': [4, 5, 6], 'B': [4, 1, 1]}) >>> df.nunique() A 3 B 2 dtype: int64
>>> df.nunique(axis=1) 0 1 1 2 2 2 dtype: int64
- pivot(index=None, columns=None, values=None)[source]¶
Return reshaped DataFrame organized by given index / column values.
Reshape data (produce a “pivot” table) based on column values. Uses unique values from specified index / columns to form axes of the resulting DataFrame. This function does not support data aggregation, multiple values will result in a MultiIndex in the columns. See the User Guide for more on reshaping.
- Parameters
index (str or object or a list of str, optional) – Column to use to make new frame’s index. If None, uses existing index.
Changed in version 1.1.0: Also accept list of index names.
columns (str or object or a list of str) – Column to use to make new frame’s columns.
Changed in version 1.1.0: Also accept list of columns names.
values (str, object or a list of the previous, optional) – Column(s) to use for populating new frame’s values. If not specified, all remaining columns will be used and the result will have hierarchically indexed columns.
- Returns
Returns reshaped DataFrame.
- Return type
- Raises
ValueError: – When there are any index, columns combinations with multiple values. DataFrame.pivot_table when you need to aggregate.
See also
DataFrame.pivot_table
Generalization of pivot that can handle duplicate values for one index/column pair.
DataFrame.unstack
Pivot based on the index values instead of a column.
wide_to_long
Wide panel to long format. Less flexible but more user-friendly than melt.
Notes
For finer-tuned control, see hierarchical indexing documentation along with the related stack/unstack methods.
Reference the user guide for more examples.
Examples
>>> df = pd.DataFrame({'foo': ['one', 'one', 'one', 'two', 'two', ... 'two'], ... 'bar': ['A', 'B', 'C', 'A', 'B', 'C'], ... 'baz': [1, 2, 3, 4, 5, 6], ... 'zoo': ['x', 'y', 'z', 'q', 'w', 't']}) >>> df foo bar baz zoo 0 one A 1 x 1 one B 2 y 2 one C 3 z 3 two A 4 q 4 two B 5 w 5 two C 6 t
>>> df.pivot(index='foo', columns='bar', values='baz') bar A B C foo one 1 2 3 two 4 5 6
>>> df.pivot(index='foo', columns='bar')['baz'] bar A B C foo one 1 2 3 two 4 5 6
>>> df.pivot(index='foo', columns='bar', values=['baz', 'zoo']) baz zoo bar A B C A B C foo one 1 2 3 x y z two 4 5 6 q w t
You could also assign a list of column names or a list of index names.
>>> df = pd.DataFrame({ ... "lev1": [1, 1, 1, 2, 2, 2], ... "lev2": [1, 1, 2, 1, 1, 2], ... "lev3": [1, 2, 1, 2, 1, 2], ... "lev4": [1, 2, 3, 4, 5, 6], ... "values": [0, 1, 2, 3, 4, 5]}) >>> df lev1 lev2 lev3 lev4 values 0 1 1 1 1 0 1 1 1 2 2 1 2 1 2 1 3 2 3 2 1 2 4 3 4 2 1 1 5 4 5 2 2 2 6 5
>>> df.pivot(index="lev1", columns=["lev2", "lev3"],values="values") lev2 1 2 lev3 1 2 1 2 lev1 1 0.0 1.0 2.0 NaN 2 4.0 3.0 NaN 5.0
>>> df.pivot(index=["lev1", "lev2"], columns=["lev3"],values="values") lev3 1 2 lev1 lev2 1 1 0.0 1.0 2 2.0 NaN 2 1 4.0 3.0 2 NaN 5.0
A ValueError is raised if there are any duplicates.
>>> df = pd.DataFrame({"foo": ['one', 'one', 'two', 'two'], ... "bar": ['A', 'A', 'B', 'C'], ... "baz": [1, 2, 3, 4]}) >>> df foo bar baz 0 one A 1 1 one A 2 2 two B 3 3 two C 4
Notice that the first two rows are the same for our index and columns arguments.
>>> df.pivot(index='foo', columns='bar', values='baz') Traceback (most recent call last): ... ValueError: Index contains duplicate entries, cannot reshape
- pivot_table(values=None, index=None, columns=None, aggfunc='mean', fill_value=None, margins=False, dropna=True, margins_name='All', observed=False, sort=True)[source]¶
Create a spreadsheet-style pivot table as a DataFrame.
The levels in the pivot table will be stored in MultiIndex objects (hierarchical indexes) on the index and columns of the result DataFrame.
- Parameters
values (column to aggregate, optional)
index (column, Grouper, array, or list of the previous) – If an array is passed, it must be the same length as the data. The list can contain any of the other types (except list). Keys to group by on the pivot table index. If an array is passed, it is being used as the same manner as column values.
columns (column, Grouper, array, or list of the previous) – If an array is passed, it must be the same length as the data. The list can contain any of the other types (except list). Keys to group by on the pivot table column. If an array is passed, it is being used as the same manner as column values.
aggfunc (function, list of functions, dict, default numpy.mean) – If list of functions passed, the resulting pivot table will have hierarchical columns whose top level are the function names (inferred from the function objects themselves) If dict is passed, the key is column to aggregate and value is function or list of functions.
fill_value (scalar, default None) – Value to replace missing values with (in the resulting pivot table, after aggregation).
margins (bool, default False) – Add all row / columns (e.g. for subtotal / grand totals).
dropna (bool, default True) – Do not include columns whose entries are all NaN.
margins_name (str, default ‘All’) – Name of the row / column that will contain the totals when margins is True.
observed (bool, default False) – This only applies if any of the groupers are Categoricals. If True: only show observed values for categorical groupers. If False: show all values for categorical groupers.
Changed in version 0.25.0.
sort (bool, default True) – Specifies if the result should be sorted.
New in version 1.3.0.
- Returns
An Excel style pivot table.
- Return type
See also
DataFrame.pivot
Pivot without aggregation that can handle non-numeric data.
DataFrame.melt
Unpivot a DataFrame from wide to long format, optionally leaving identifiers set.
wide_to_long
Wide panel to long format. Less flexible but more user-friendly than melt.
Notes
Reference the user guide for more examples.
Examples
>>> df = pd.DataFrame({"A": ["foo", "foo", "foo", "foo", "foo", ... "bar", "bar", "bar", "bar"], ... "B": ["one", "one", "one", "two", "two", ... "one", "one", "two", "two"], ... "C": ["small", "large", "large", "small", ... "small", "large", "small", "small", ... "large"], ... "D": [1, 2, 2, 3, 3, 4, 5, 6, 7], ... "E": [2, 4, 5, 5, 6, 6, 8, 9, 9]}) >>> df A B C D E 0 foo one small 1 2 1 foo one large 2 4 2 foo one large 2 5 3 foo two small 3 5 4 foo two small 3 6 5 bar one large 4 6 6 bar one small 5 8 7 bar two small 6 9 8 bar two large 7 9
This first example aggregates values by taking the sum.
>>> table = pd.pivot_table(df, values='D', index=['A', 'B'], ... columns=['C'], aggfunc=np.sum) >>> table C large small A B bar one 4.0 5.0 two 7.0 6.0 foo one 4.0 1.0 two NaN 6.0
We can also fill missing values using the fill_value parameter.
>>> table = pd.pivot_table(df, values='D', index=['A', 'B'], ... columns=['C'], aggfunc=np.sum, fill_value=0) >>> table C large small A B bar one 4 5 two 7 6 foo one 4 1 two 0 6
The next example aggregates by taking the mean across multiple columns.
>>> table = pd.pivot_table(df, values=['D', 'E'], index=['A', 'C'], ... aggfunc={'D': np.mean, ... 'E': np.mean}) >>> table D E A C bar large 5.500000 7.500000 small 5.500000 8.500000 foo large 2.000000 4.500000 small 2.333333 4.333333
We can also calculate multiple types of aggregations for any given value column.
>>> table = pd.pivot_table(df, values=['D', 'E'], index=['A', 'C'], ... aggfunc={'D': np.mean, ... 'E': [min, max, np.mean]}) >>> table D E mean max mean min A C bar large 5.500000 9 7.500000 6 small 5.500000 9 8.500000 8 foo large 2.000000 5 4.500000 4 small 2.333333 6 4.333333 2
- plot¶
alias of
pandas.plotting._core.PlotAccessor
- pop(item)[source]¶
Return item and drop from frame. Raise KeyError if not found.
- Parameters
item (label) – Label of column to be popped.
- Return type
Examples
>>> df = pd.DataFrame([('falcon', 'bird', 389.0), ... ('parrot', 'bird', 24.0), ... ('lion', 'mammal', 80.5), ... ('monkey', 'mammal', np.nan)], ... columns=('name', 'class', 'max_speed')) >>> df name class max_speed 0 falcon bird 389.0 1 parrot bird 24.0 2 lion mammal 80.5 3 monkey mammal NaN
>>> df.pop('class') 0 bird 1 bird 2 mammal 3 mammal Name: class, dtype: object
>>> df name max_speed 0 falcon 389.0 1 parrot 24.0 2 lion 80.5 3 monkey NaN
- pow(other, axis='columns', level=None, fill_value=None)¶
Get Exponential power of dataframe and other, element-wise (binary operator pow).
Equivalent to
dataframe ** other
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, rpow.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- prod(axis=None, skipna=True, level=None, numeric_only=None, min_count=0, **kwargs)¶
Return the product of the values over the requested axis.
- Parameters
axis ({index (0), columns (1)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a Series.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
min_count (int, default 0) – The required number of valid values to perform the operation. If fewer than
min_count
non-NA values are present the result will be NA.**kwargs – Additional keyword arguments to be passed to the function.
- Return type
See also
Series.sum
Return the sum.
Series.min
Return the minimum.
Series.max
Return the maximum.
Series.idxmin
Return the index of the minimum.
Series.idxmax
Return the index of the maximum.
DataFrame.sum
Return the sum over the requested axis.
DataFrame.min
Return the minimum over the requested axis.
DataFrame.max
Return the maximum over the requested axis.
DataFrame.idxmin
Return the index of the minimum over the requested axis.
DataFrame.idxmax
Return the index of the maximum over the requested axis.
Examples
By default, the product of an empty or all-NA Series is
1
>>> pd.Series([], dtype="float64").prod() 1.0
This can be controlled with the
min_count
parameter>>> pd.Series([], dtype="float64").prod(min_count=1) nan
Thanks to the
skipna
parameter,min_count
handles all-NA and empty series identically.>>> pd.Series([np.nan]).prod() 1.0
>>> pd.Series([np.nan]).prod(min_count=1) nan
- product(axis=None, skipna=True, level=None, numeric_only=None, min_count=0, **kwargs)¶
Return the product of the values over the requested axis.
- Parameters
axis ({index (0), columns (1)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a Series.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
min_count (int, default 0) – The required number of valid values to perform the operation. If fewer than
min_count
non-NA values are present the result will be NA.**kwargs – Additional keyword arguments to be passed to the function.
- Return type
See also
Series.sum
Return the sum.
Series.min
Return the minimum.
Series.max
Return the maximum.
Series.idxmin
Return the index of the minimum.
Series.idxmax
Return the index of the maximum.
DataFrame.sum
Return the sum over the requested axis.
DataFrame.min
Return the minimum over the requested axis.
DataFrame.max
Return the maximum over the requested axis.
DataFrame.idxmin
Return the index of the minimum over the requested axis.
DataFrame.idxmax
Return the index of the maximum over the requested axis.
Examples
By default, the product of an empty or all-NA Series is
1
>>> pd.Series([], dtype="float64").prod() 1.0
This can be controlled with the
min_count
parameter>>> pd.Series([], dtype="float64").prod(min_count=1) nan
Thanks to the
skipna
parameter,min_count
handles all-NA and empty series identically.>>> pd.Series([np.nan]).prod() 1.0
>>> pd.Series([np.nan]).prod(min_count=1) nan
- quantile(q=0.5, axis=0, numeric_only=True, interpolation='linear')[source]¶
Return values at the given quantile over requested axis.
- Parameters
q (float or array-like, default 0.5 (50% quantile)) – Value between 0 <= q <= 1, the quantile(s) to compute.
axis ({0, 1, ‘index’, ‘columns’}, default 0) – Equals 0 or ‘index’ for row-wise, 1 or ‘columns’ for column-wise.
numeric_only (bool, default True) – If False, the quantile of datetime and timedelta data will be computed as well.
interpolation ({‘linear’, ‘lower’, ‘higher’, ‘midpoint’, ‘nearest’}) – This optional parameter specifies the interpolation method to use, when the desired quantile lies between two data points i and j:
linear: i + (j - i) * fraction, where fraction is the fractional part of the index surrounded by i and j.
lower: i.
higher: j.
nearest: i or j whichever is nearest.
midpoint: (i + j) / 2.
- Returns
- If
q
is an array, a DataFrame will be returned where the index is
q
, the columns are the columns of self, and the values are the quantiles.- If
q
is a float, a Series will be returned where the index is the columns of self and the values are the quantiles.
- If
- Return type
See also
core.window.Rolling.quantile
Rolling quantile.
numpy.percentile
Numpy function to compute the percentile.
Examples
>>> df = pd.DataFrame(np.array([[1, 1], [2, 10], [3, 100], [4, 100]]), ... columns=['a', 'b']) >>> df.quantile(.1) a 1.3 b 3.7 Name: 0.1, dtype: float64 >>> df.quantile([.1, .5]) a b 0.1 1.3 3.7 0.5 2.5 55.0
Specifying numeric_only=False will also compute the quantile of datetime and timedelta data.
>>> df = pd.DataFrame({'A': [1, 2], ... 'B': [pd.Timestamp('2010'), ... pd.Timestamp('2011')], ... 'C': [pd.Timedelta('1 days'), ... pd.Timedelta('2 days')]}) >>> df.quantile(0.5, numeric_only=False) A 1.5 B 2010-07-02 12:00:00 C 1 days 12:00:00 Name: 0.5, dtype: object
- query(expr, inplace=False, **kwargs)[source]¶
Query the columns of a DataFrame with a boolean expression.
- Parameters
expr (str) – The query string to evaluate.
You can refer to variables in the environment by prefixing them with an ‘@’ character like
@a + b
.You can refer to column names that are not valid Python variable names by surrounding them in backticks. Thus, column names containing spaces or punctuations (besides underscores) or starting with digits must be surrounded by backticks. (For example, a column named “Area (cm^2)” would be referenced as
`Area (cm^2)`
). Column names which are Python keywords (like “list”, “for”, “import”, etc) cannot be used.For example, if one of your columns is called
a a
and you want to sum it withb
, your query should be`a a` + b
.New in version 0.25.0: Backtick quoting introduced.
New in version 1.0.0: Expanding functionality of backtick quoting for more than only spaces.
inplace (bool) – Whether the query should modify the data in place or return a modified copy.
**kwargs – See the documentation for
eval()
for complete details on the keyword arguments accepted byDataFrame.query()
.
- Returns
DataFrame resulting from the provided query expression or None if
inplace=True
.- Return type
DataFrame or None
See also
eval
Evaluate a string describing operations on DataFrame columns.
DataFrame.eval
Evaluate a string describing operations on DataFrame columns.
Notes
The result of the evaluation of this expression is first passed to
DataFrame.loc
and if that fails because of a multidimensional key (e.g., a DataFrame) then the result will be passed toDataFrame.__getitem__()
.This method uses the top-level
eval()
function to evaluate the passed query.The
query()
method uses a slightly modified Python syntax by default. For example, the&
and|
(bitwise) operators have the precedence of their boolean cousins,and
andor
. This is syntactically valid Python, however the semantics are different.You can change the semantics of the expression by passing the keyword argument
parser='python'
. This enforces the same semantics as evaluation in Python space. Likewise, you can passengine='python'
to evaluate an expression using Python itself as a backend. This is not recommended as it is inefficient compared to usingnumexpr
as the engine.The
DataFrame.index
andDataFrame.columns
attributes of theDataFrame
instance are placed in the query namespace by default, which allows you to treat both the index and columns of the frame as a column in the frame. The identifierindex
is used for the frame index; you can also use the name of the index to identify it in a query. Please note that Python keywords may not be used as identifiers.For further details and examples see the
query
documentation in indexing.Backtick quoted variables
Backtick quoted variables are parsed as literal Python code and are converted internally to a Python valid identifier. This can lead to the following problems.
During parsing a number of disallowed characters inside the backtick quoted string are replaced by strings that are allowed as a Python identifier. These characters include all operators in Python, the space character, the question mark, the exclamation mark, the dollar sign, and the euro sign. For other characters that fall outside the ASCII range (U+0001..U+007F) and those that are not further specified in PEP 3131, the query parser will raise an error. This excludes whitespace different than the space character, but also the hashtag (as it is used for comments) and the backtick itself (backtick can also not be escaped).
In a special case, quotes that make a pair around a backtick can confuse the parser. For example,
`it's` > `that's`
will raise an error, as it forms a quoted string ('s > `that'
) with a backtick inside.See also the Python documentation about lexical analysis (https://docs.python.org/3/reference/lexical_analysis.html) in combination with the source code in
pandas.core.computation.parsing
.Examples
>>> df = pd.DataFrame({'A': range(1, 6), ... 'B': range(10, 0, -2), ... 'C C': range(10, 5, -1)}) >>> df A B C C 0 1 10 10 1 2 8 9 2 3 6 8 3 4 4 7 4 5 2 6 >>> df.query('A > B') A B C C 4 5 2 6
The previous expression is equivalent to
>>> df[df.A > df.B] A B C C 4 5 2 6
For columns with spaces in their name, you can use backtick quoting.
>>> df.query('B == `C C`') A B C C 0 1 10 10
The previous expression is equivalent to
>>> df[df.B == df['C C']] A B C C 0 1 10 10
- radd(other, axis='columns', level=None, fill_value=None)¶
Get Addition of dataframe and other, element-wise (binary operator radd).
Equivalent to
other + dataframe
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, add.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- rdiv(other, axis='columns', level=None, fill_value=None)¶
Get Floating division of dataframe and other, element-wise (binary operator rtruediv).
Equivalent to
other / dataframe
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, truediv.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- reindex(labels=None, index=None, columns=None, axis=None, method=None, copy=True, level=None, fill_value=nan, limit=None, tolerance=None)[source]¶
Conform Series/DataFrame to new index with optional filling logic.
Places NA/NaN in locations having no value in the previous index. A new object is produced unless the new index is equivalent to the current one and
copy=False
.- Parameters
keywords for axes (array-like, optional) – New labels / index to conform to, should be specified using keywords. Preferably an Index object to avoid duplicating data.
method ({None, ‘backfill’/’bfill’, ‘pad’/’ffill’, ‘nearest’}) – Method to use for filling holes in reindexed DataFrame. Please note: this is only applicable to DataFrames/Series with a monotonically increasing/decreasing index.
None (default): don’t fill gaps
pad / ffill: Propagate last valid observation forward to next valid.
backfill / bfill: Use next valid observation to fill gap.
nearest: Use nearest valid observations to fill gap.
copy (bool, default True) – Return a new object, even if the passed indexes are the same.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (scalar, default np.NaN) – Value to use for missing values. Defaults to NaN, but can be any “compatible” value.
limit (int, default None) – Maximum number of consecutive elements to forward or backward fill.
tolerance (optional) – Maximum distance between original and new labels for inexact matches. The values of the index at the matching locations most satisfy the equation
abs(index[indexer] - target) <= tolerance
.Tolerance may be a scalar value, which applies the same tolerance to all values, or list-like, which applies variable tolerance per element. List-like includes list, tuple, array, Series, and must be the same size as the index and its dtype must exactly match the index’s type.
- Return type
Series/DataFrame with changed index.
See also
DataFrame.set_index
Set row labels.
DataFrame.reset_index
Remove row labels or move them to new columns.
DataFrame.reindex_like
Change to same indices as other DataFrame.
Examples
DataFrame.reindex
supports two calling conventions(index=index_labels, columns=column_labels, ...)
(labels, axis={'index', 'columns'}, ...)
We highly recommend using keyword arguments to clarify your intent.
Create a dataframe with some fictional data.
>>> index = ['Firefox', 'Chrome', 'Safari', 'IE10', 'Konqueror'] >>> df = pd.DataFrame({'http_status': [200, 200, 404, 404, 301], ... 'response_time': [0.04, 0.02, 0.07, 0.08, 1.0]}, ... index=index) >>> df http_status response_time Firefox 200 0.04 Chrome 200 0.02 Safari 404 0.07 IE10 404 0.08 Konqueror 301 1.00
Create a new index and reindex the dataframe. By default values in the new index that do not have corresponding records in the dataframe are assigned
NaN
.>>> new_index = ['Safari', 'Iceweasel', 'Comodo Dragon', 'IE10', ... 'Chrome'] >>> df.reindex(new_index) http_status response_time Safari 404.0 0.07 Iceweasel NaN NaN Comodo Dragon NaN NaN IE10 404.0 0.08 Chrome 200.0 0.02
We can fill in the missing values by passing a value to the keyword
fill_value
. Because the index is not monotonically increasing or decreasing, we cannot use arguments to the keywordmethod
to fill theNaN
values.>>> df.reindex(new_index, fill_value=0) http_status response_time Safari 404 0.07 Iceweasel 0 0.00 Comodo Dragon 0 0.00 IE10 404 0.08 Chrome 200 0.02
>>> df.reindex(new_index, fill_value='missing') http_status response_time Safari 404 0.07 Iceweasel missing missing Comodo Dragon missing missing IE10 404 0.08 Chrome 200 0.02
We can also reindex the columns.
>>> df.reindex(columns=['http_status', 'user_agent']) http_status user_agent Firefox 200 NaN Chrome 200 NaN Safari 404 NaN IE10 404 NaN Konqueror 301 NaN
Or we can use “axis-style” keyword arguments
>>> df.reindex(['http_status', 'user_agent'], axis="columns") http_status user_agent Firefox 200 NaN Chrome 200 NaN Safari 404 NaN IE10 404 NaN Konqueror 301 NaN
To further illustrate the filling functionality in
reindex
, we will create a dataframe with a monotonically increasing index (for example, a sequence of dates).>>> date_index = pd.date_range('1/1/2010', periods=6, freq='D') >>> df2 = pd.DataFrame({"prices": [100, 101, np.nan, 100, 89, 88]}, ... index=date_index) >>> df2 prices 2010-01-01 100.0 2010-01-02 101.0 2010-01-03 NaN 2010-01-04 100.0 2010-01-05 89.0 2010-01-06 88.0
Suppose we decide to expand the dataframe to cover a wider date range.
>>> date_index2 = pd.date_range('12/29/2009', periods=10, freq='D') >>> df2.reindex(date_index2) prices 2009-12-29 NaN 2009-12-30 NaN 2009-12-31 NaN 2010-01-01 100.0 2010-01-02 101.0 2010-01-03 NaN 2010-01-04 100.0 2010-01-05 89.0 2010-01-06 88.0 2010-01-07 NaN
The index entries that did not have a value in the original data frame (for example, ‘2009-12-29’) are by default filled with
NaN
. If desired, we can fill in the missing values using one of several options.For example, to back-propagate the last valid value to fill the
NaN
values, passbfill
as an argument to themethod
keyword.>>> df2.reindex(date_index2, method='bfill') prices 2009-12-29 100.0 2009-12-30 100.0 2009-12-31 100.0 2010-01-01 100.0 2010-01-02 101.0 2010-01-03 NaN 2010-01-04 100.0 2010-01-05 89.0 2010-01-06 88.0 2010-01-07 NaN
Please note that the
NaN
value present in the original dataframe (at index value 2010-01-03) will not be filled by any of the value propagation schemes. This is because filling while reindexing does not look at dataframe values, but only compares the original and desired indexes. If you do want to fill in theNaN
values present in the original dataframe, use thefillna()
method.See the user guide for more.
- rename(mapper=None, *, index=None, columns=None, axis=None, copy=True, inplace=False, level=None, errors='ignore')[source]¶
Alter axes labels.
Function / dict values must be unique (1-to-1). Labels not contained in a dict / Series will be left as-is. Extra labels listed don’t throw an error.
See the user guide for more.
- Parameters
mapper (dict-like or function) – Dict-like or function transformations to apply to that axis’ values. Use either
mapper
andaxis
to specify the axis to target withmapper
, orindex
andcolumns
.index (dict-like or function) – Alternative to specifying axis (
mapper, axis=0
is equivalent toindex=mapper
).columns (dict-like or function) – Alternative to specifying axis (
mapper, axis=1
is equivalent tocolumns=mapper
).axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – Axis to target with
mapper
. Can be either the axis name (‘index’, ‘columns’) or number (0, 1). The default is ‘index’.copy (bool, default True) – Also copy underlying data.
inplace (bool, default False) – Whether to return a new DataFrame. If True then value of copy is ignored.
level (int or level name, default None) – In case of a MultiIndex, only rename labels in the specified level.
errors ({‘ignore’, ‘raise’}, default ‘ignore’) – If ‘raise’, raise a KeyError when a dict-like mapper, index, or columns contains labels that are not present in the Index being transformed. If ‘ignore’, existing keys will be renamed and extra keys will be ignored.
- Returns
DataFrame with the renamed axis labels or None if
inplace=True
.- Return type
DataFrame or None
- Raises
KeyError – If any of the labels is not found in the selected axis and “errors=’raise’”.
See also
DataFrame.rename_axis
Set the name of the axis.
Examples
DataFrame.rename
supports two calling conventions(index=index_mapper, columns=columns_mapper, ...)
(mapper, axis={'index', 'columns'}, ...)
We highly recommend using keyword arguments to clarify your intent.
Rename columns using a mapping:
>>> df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]}) >>> df.rename(columns={"A": "a", "B": "c"}) a c 0 1 4 1 2 5 2 3 6
Rename index using a mapping:
>>> df.rename(index={0: "x", 1: "y", 2: "z"}) A B x 1 4 y 2 5 z 3 6
Cast index labels to a different type:
>>> df.index RangeIndex(start=0, stop=3, step=1) >>> df.rename(index=str).index Index(['0', '1', '2'], dtype='object')
>>> df.rename(columns={"A": "a", "B": "b", "C": "c"}, errors="raise") Traceback (most recent call last): KeyError: ['C'] not found in axis
Using axis-style parameters:
>>> df.rename(str.lower, axis='columns') a b 0 1 4 1 2 5 2 3 6
>>> df.rename({1: 2, 2: 4}, axis='index') A B 0 1 4 2 2 5 4 3 6
- reorder_levels(order, axis=0)[source]¶
Rearrange index levels using input order. May not drop or duplicate levels.
- Parameters
order (list of int or list of str) – List representing new level order. Reference level by number (position) or by key (label).
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – Where to reorder levels.
- Return type
Examples
>>> data = { ... "class": ["Mammals", "Mammals", "Reptiles"], ... "diet": ["Omnivore", "Carnivore", "Carnivore"], ... "species": ["Humans", "Dogs", "Snakes"], ... } >>> df = pd.DataFrame(data, columns=["class", "diet", "species"]) >>> df = df.set_index(["class", "diet"]) >>> df species class diet Mammals Omnivore Humans Carnivore Dogs Reptiles Carnivore Snakes
Let’s reorder the levels of the index:
>>> df.reorder_levels(["diet", "class"]) species diet class Omnivore Mammals Humans Carnivore Mammals Dogs Reptiles Snakes
- replace(to_replace=None, value=NoDefault.no_default, inplace=False, limit=None, regex=False, method=NoDefault.no_default)[source]¶
Replace values given in to_replace with value.
Values of the DataFrame are replaced with other values dynamically.
This differs from updating with
.loc
or.iloc
, which require you to specify a location to update with some value.- Parameters
to_replace (str, regex, list, dict, Series, int, float, or None) – How to find the values that will be replaced.
numeric, str or regex:
numeric: numeric values equal to to_replace will be replaced with value
str: string exactly matching to_replace will be replaced with value
regex: regexs matching to_replace will be replaced with value
list of str, regex, or numeric:
First, if to_replace and value are both lists, they must be the same length.
Second, if
regex=True
then all of the strings in both lists will be interpreted as regexs otherwise they will match directly. This doesn’t matter much for value since there are only a few possible substitution regexes you can use.str, regex and numeric rules apply as above.
dict:
Dicts can be used to specify different replacement values for different existing values. For example,
{'a': 'b', 'y': 'z'}
replaces the value ‘a’ with ‘b’ and ‘y’ with ‘z’. To use a dict in this way the value parameter should be None.For a DataFrame a dict can specify that different values should be replaced in different columns. For example,
{'a': 1, 'b': 'z'}
looks for the value 1 in column ‘a’ and the value ‘z’ in column ‘b’ and replaces these values with whatever is specified in value. The value parameter should not beNone
in this case. You can treat this as a special case of passing two lists except that you are specifying the column to search in.For a DataFrame nested dictionaries, e.g.,
{'a': {'b': np.nan}}
, are read as follows: look in column ‘a’ for the value ‘b’ and replace it with NaN. The value parameter should beNone
to use a nested dict in this way. You can nest regular expressions as well. Note that column names (the top-level dictionary keys in a nested dictionary) cannot be regular expressions.
None:
This means that the regex argument must be a string, compiled regular expression, or list, dict, ndarray or Series of such elements. If value is also
None
then this must be a nested dictionary or Series.
See the examples section for examples of each of these.
value (scalar, dict, list, str, regex, default None) – Value to replace any values matching to_replace with. For a DataFrame a dict of values can be used to specify which value to use for each column (columns not in the dict will not be filled). Regular expressions, strings and lists or dicts of such objects are also allowed.
inplace (bool, default False) – If True, performs operation inplace and returns None.
limit (int, default None) – Maximum size gap to forward or backward fill.
regex (bool or same types as to_replace, default False) – Whether to interpret to_replace and/or value as regular expressions. If this is
True
then to_replace must be a string. Alternatively, this could be a regular expression or a list, dict, or array of regular expressions in which case to_replace must beNone
.method ({‘pad’, ‘ffill’, ‘bfill’, None}) – The method to use when for replacement, when to_replace is a scalar, list or tuple and value is
None
.Changed in version 0.23.0: Added to DataFrame.
- Returns
Object after replacement.
- Return type
- Raises
If regex is not a
bool
and to_replace is notNone
.
If to_replace is not a scalar, array-like,
dict
, orNone
* If to_replace is adict
and value is not alist
,dict
,ndarray
, orSeries
* If to_replace isNone
and regex is not compilable into a regular expression or is a list, dict, ndarray, or Series. * When replacing multiplebool
ordatetime64
objects and the arguments to to_replace does not match the type of the value being replaced
If a
list
or anndarray
is passed to to_replace and value but they are not the same length.
See also
DataFrame.fillna
Fill NA values.
DataFrame.where
Replace values based on boolean condition.
Series.str.replace
Simple string replacement.
Notes
Regex substitution is performed under the hood with
re.sub
. The rules for substitution forre.sub
are the same.Regular expressions will only substitute on strings, meaning you cannot provide, for example, a regular expression matching floating point numbers and expect the columns in your frame that have a numeric dtype to be matched. However, if those floating point numbers are strings, then you can do this.
This method has a lot of options. You are encouraged to experiment and play with this method to gain intuition about how it works.
When dict is used as the to_replace value, it is like key(s) in the dict are the to_replace part and value(s) in the dict are the value parameter.
Examples
Scalar `to_replace` and `value`
>>> s = pd.Series([1, 2, 3, 4, 5]) >>> s.replace(1, 5) 0 5 1 2 2 3 3 4 4 5 dtype: int64
>>> df = pd.DataFrame({'A': [0, 1, 2, 3, 4], ... 'B': [5, 6, 7, 8, 9], ... 'C': ['a', 'b', 'c', 'd', 'e']}) >>> df.replace(0, 5) A B C 0 5 5 a 1 1 6 b 2 2 7 c 3 3 8 d 4 4 9 e
List-like `to_replace`
>>> df.replace([0, 1, 2, 3], 4) A B C 0 4 5 a 1 4 6 b 2 4 7 c 3 4 8 d 4 4 9 e
>>> df.replace([0, 1, 2, 3], [4, 3, 2, 1]) A B C 0 4 5 a 1 3 6 b 2 2 7 c 3 1 8 d 4 4 9 e
>>> s.replace([1, 2], method='bfill') 0 3 1 3 2 3 3 4 4 5 dtype: int64
dict-like `to_replace`
>>> df.replace({0: 10, 1: 100}) A B C 0 10 5 a 1 100 6 b 2 2 7 c 3 3 8 d 4 4 9 e
>>> df.replace({'A': 0, 'B': 5}, 100) A B C 0 100 100 a 1 1 6 b 2 2 7 c 3 3 8 d 4 4 9 e
>>> df.replace({'A': {0: 100, 4: 400}}) A B C 0 100 5 a 1 1 6 b 2 2 7 c 3 3 8 d 4 400 9 e
Regular expression `to_replace`
>>> df = pd.DataFrame({'A': ['bat', 'foo', 'bait'], ... 'B': ['abc', 'bar', 'xyz']}) >>> df.replace(to_replace=r'^ba.$', value='new', regex=True) A B 0 new abc 1 foo new 2 bait xyz
>>> df.replace({'A': r'^ba.$'}, {'A': 'new'}, regex=True) A B 0 new abc 1 foo bar 2 bait xyz
>>> df.replace(regex=r'^ba.$', value='new') A B 0 new abc 1 foo new 2 bait xyz
>>> df.replace(regex={r'^ba.$': 'new', 'foo': 'xyz'}) A B 0 new abc 1 xyz new 2 bait xyz
>>> df.replace(regex=[r'^ba.$', 'foo'], value='new') A B 0 new abc 1 new new 2 bait xyz
Compare the behavior of
s.replace({'a': None})
ands.replace('a', None)
to understand the peculiarities of the to_replace parameter:>>> s = pd.Series([10, 'a', 'a', 'b', 'a'])
When one uses a dict as the to_replace value, it is like the value(s) in the dict are equal to the value parameter.
s.replace({'a': None})
is equivalent tos.replace(to_replace={'a': None}, value=None, method=None)
:>>> s.replace({'a': None}) 0 10 1 None 2 None 3 b 4 None dtype: object
When
value
is not explicitly passed and to_replace is a scalar, list or tuple, replace uses the method parameter (default ‘pad’) to do the replacement. So this is why the ‘a’ values are being replaced by 10 in rows 1 and 2 and ‘b’ in row 4 in this case.>>> s.replace('a') 0 10 1 10 2 10 3 b 4 b dtype: object
On the other hand, if
None
is explicitly passed forvalue
, it will be respected:>>> s.replace('a', None) 0 10 1 None 2 None 3 b 4 None dtype: object
Changed in version 1.4.0: Previously the explicit
None
was silently ignored.
- resample(rule, axis=0, closed=None, label=None, convention='start', kind=None, loffset=None, base=None, on=None, level=None, origin='start_day', offset=None)[source]¶
Resample time-series data.
Convenience method for frequency conversion and resampling of time series. The object must have a datetime-like index (DatetimeIndex, PeriodIndex, or TimedeltaIndex), or the caller must pass the label of a datetime-like series/index to the
on
/level
keyword parameter.- Parameters
rule (DateOffset, Timedelta or str) – The offset string or object representing target conversion.
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – Which axis to use for up- or down-sampling. For Series this will default to 0, i.e. along the rows. Must be DatetimeIndex, TimedeltaIndex or PeriodIndex.
closed ({‘right’, ‘left’}, default None) – Which side of bin interval is closed. The default is ‘left’ for all frequency offsets except for ‘M’, ‘A’, ‘Q’, ‘BM’, ‘BA’, ‘BQ’, and ‘W’ which all have a default of ‘right’.
label ({‘right’, ‘left’}, default None) – Which bin edge label to label bucket with. The default is ‘left’ for all frequency offsets except for ‘M’, ‘A’, ‘Q’, ‘BM’, ‘BA’, ‘BQ’, and ‘W’ which all have a default of ‘right’.
convention ({‘start’, ‘end’, ‘s’, ‘e’}, default ‘start’) – For PeriodIndex only, controls whether to use the start or end of rule.
kind ({‘timestamp’, ‘period’}, optional, default None) – Pass ‘timestamp’ to convert the resulting index to a DateTimeIndex or ‘period’ to convert it to a PeriodIndex. By default the input representation is retained.
loffset (timedelta, default None) – Adjust the resampled time labels.
Deprecated since version 1.1.0: You should add the loffset to the df.index after the resample. See below.
base (int, default 0) – For frequencies that evenly subdivide 1 day, the “origin” of the aggregated intervals. For example, for ‘5min’ frequency, base could range from 0 through 4. Defaults to 0.
Deprecated since version 1.1.0: The new arguments that you should use are ‘offset’ or ‘origin’.
on (str, optional) – For a DataFrame, column to use instead of index for resampling. Column must be datetime-like.
level (str or int, optional) – For a MultiIndex, level (name or number) to use for resampling. level must be datetime-like.
origin (Timestamp or str, default ‘start_day’) – The timestamp on which to adjust the grouping. The timezone of origin must match the timezone of the index. If string, must be one of the following:
‘epoch’: origin is 1970-01-01
‘start’: origin is the first value of the timeseries
‘start_day’: origin is the first day at midnight of the timeseries
New in version 1.1.0.
‘end’: origin is the last value of the timeseries
‘end_day’: origin is the ceiling midnight of the last day
New in version 1.3.0.
offset (Timedelta or str, default is None) – An offset timedelta added to the origin.
New in version 1.1.0.
- Returns
Resampler
object.- Return type
pandas.core.Resampler
See also
Series.resample
Resample a Series.
DataFrame.resample
Resample a DataFrame.
groupby
Group DataFrame by mapping, function, label, or list of labels.
asfreq
Reindex a DataFrame with the given frequency without grouping.
Notes
See the user guide for more.
To learn more about the offset strings, please see this link.
Examples
Start by creating a series with 9 one minute timestamps.
>>> index = pd.date_range('1/1/2000', periods=9, freq='T') >>> series = pd.Series(range(9), index=index) >>> series 2000-01-01 00:00:00 0 2000-01-01 00:01:00 1 2000-01-01 00:02:00 2 2000-01-01 00:03:00 3 2000-01-01 00:04:00 4 2000-01-01 00:05:00 5 2000-01-01 00:06:00 6 2000-01-01 00:07:00 7 2000-01-01 00:08:00 8 Freq: T, dtype: int64
Downsample the series into 3 minute bins and sum the values of the timestamps falling into a bin.
>>> series.resample('3T').sum() 2000-01-01 00:00:00 3 2000-01-01 00:03:00 12 2000-01-01 00:06:00 21 Freq: 3T, dtype: int64
Downsample the series into 3 minute bins as above, but label each bin using the right edge instead of the left. Please note that the value in the bucket used as the label is not included in the bucket, which it labels. For example, in the original series the bucket
2000-01-01 00:03:00
contains the value 3, but the summed value in the resampled bucket with the label2000-01-01 00:03:00
does not include 3 (if it did, the summed value would be 6, not 3). To include this value close the right side of the bin interval as illustrated in the example below this one.>>> series.resample('3T', label='right').sum() 2000-01-01 00:03:00 3 2000-01-01 00:06:00 12 2000-01-01 00:09:00 21 Freq: 3T, dtype: int64
Downsample the series into 3 minute bins as above, but close the right side of the bin interval.
>>> series.resample('3T', label='right', closed='right').sum() 2000-01-01 00:00:00 0 2000-01-01 00:03:00 6 2000-01-01 00:06:00 15 2000-01-01 00:09:00 15 Freq: 3T, dtype: int64
Upsample the series into 30 second bins.
>>> series.resample('30S').asfreq()[0:5] # Select first 5 rows 2000-01-01 00:00:00 0.0 2000-01-01 00:00:30 NaN 2000-01-01 00:01:00 1.0 2000-01-01 00:01:30 NaN 2000-01-01 00:02:00 2.0 Freq: 30S, dtype: float64
Upsample the series into 30 second bins and fill the
NaN
values using thepad
method.>>> series.resample('30S').pad()[0:5] 2000-01-01 00:00:00 0 2000-01-01 00:00:30 0 2000-01-01 00:01:00 1 2000-01-01 00:01:30 1 2000-01-01 00:02:00 2 Freq: 30S, dtype: int64
Upsample the series into 30 second bins and fill the
NaN
values using thebfill
method.>>> series.resample('30S').bfill()[0:5] 2000-01-01 00:00:00 0 2000-01-01 00:00:30 1 2000-01-01 00:01:00 1 2000-01-01 00:01:30 2 2000-01-01 00:02:00 2 Freq: 30S, dtype: int64
Pass a custom function via
apply
>>> def custom_resampler(arraylike): ... return np.sum(arraylike) + 5 ... >>> series.resample('3T').apply(custom_resampler) 2000-01-01 00:00:00 8 2000-01-01 00:03:00 17 2000-01-01 00:06:00 26 Freq: 3T, dtype: int64
For a Series with a PeriodIndex, the keyword convention can be used to control whether to use the start or end of rule.
Resample a year by quarter using ‘start’ convention. Values are assigned to the first quarter of the period.
>>> s = pd.Series([1, 2], index=pd.period_range('2012-01-01', ... freq='A', ... periods=2)) >>> s 2012 1 2013 2 Freq: A-DEC, dtype: int64 >>> s.resample('Q', convention='start').asfreq() 2012Q1 1.0 2012Q2 NaN 2012Q3 NaN 2012Q4 NaN 2013Q1 2.0 2013Q2 NaN 2013Q3 NaN 2013Q4 NaN Freq: Q-DEC, dtype: float64
Resample quarters by month using ‘end’ convention. Values are assigned to the last month of the period.
>>> q = pd.Series([1, 2, 3, 4], index=pd.period_range('2018-01-01', ... freq='Q', ... periods=4)) >>> q 2018Q1 1 2018Q2 2 2018Q3 3 2018Q4 4 Freq: Q-DEC, dtype: int64 >>> q.resample('M', convention='end').asfreq() 2018-03 1.0 2018-04 NaN 2018-05 NaN 2018-06 2.0 2018-07 NaN 2018-08 NaN 2018-09 3.0 2018-10 NaN 2018-11 NaN 2018-12 4.0 Freq: M, dtype: float64
For DataFrame objects, the keyword on can be used to specify the column instead of the index for resampling.
>>> d = {'price': [10, 11, 9, 13, 14, 18, 17, 19], ... 'volume': [50, 60, 40, 100, 50, 100, 40, 50]} >>> df = pd.DataFrame(d) >>> df['week_starting'] = pd.date_range('01/01/2018', ... periods=8, ... freq='W') >>> df price volume week_starting 0 10 50 2018-01-07 1 11 60 2018-01-14 2 9 40 2018-01-21 3 13 100 2018-01-28 4 14 50 2018-02-04 5 18 100 2018-02-11 6 17 40 2018-02-18 7 19 50 2018-02-25 >>> df.resample('M', on='week_starting').mean() price volume week_starting 2018-01-31 10.75 62.5 2018-02-28 17.00 60.0
For a DataFrame with MultiIndex, the keyword level can be used to specify on which level the resampling needs to take place.
>>> days = pd.date_range('1/1/2000', periods=4, freq='D') >>> d2 = {'price': [10, 11, 9, 13, 14, 18, 17, 19], ... 'volume': [50, 60, 40, 100, 50, 100, 40, 50]} >>> df2 = pd.DataFrame( ... d2, ... index=pd.MultiIndex.from_product( ... [days, ['morning', 'afternoon']] ... ) ... ) >>> df2 price volume 2000-01-01 morning 10 50 afternoon 11 60 2000-01-02 morning 9 40 afternoon 13 100 2000-01-03 morning 14 50 afternoon 18 100 2000-01-04 morning 17 40 afternoon 19 50 >>> df2.resample('D', level=0).sum() price volume 2000-01-01 21 110 2000-01-02 22 140 2000-01-03 32 150 2000-01-04 36 90
If you want to adjust the start of the bins based on a fixed timestamp:
>>> start, end = '2000-10-01 23:30:00', '2000-10-02 00:30:00' >>> rng = pd.date_range(start, end, freq='7min') >>> ts = pd.Series(np.arange(len(rng)) * 3, index=rng) >>> ts 2000-10-01 23:30:00 0 2000-10-01 23:37:00 3 2000-10-01 23:44:00 6 2000-10-01 23:51:00 9 2000-10-01 23:58:00 12 2000-10-02 00:05:00 15 2000-10-02 00:12:00 18 2000-10-02 00:19:00 21 2000-10-02 00:26:00 24 Freq: 7T, dtype: int64
>>> ts.resample('17min').sum() 2000-10-01 23:14:00 0 2000-10-01 23:31:00 9 2000-10-01 23:48:00 21 2000-10-02 00:05:00 54 2000-10-02 00:22:00 24 Freq: 17T, dtype: int64
>>> ts.resample('17min', origin='epoch').sum() 2000-10-01 23:18:00 0 2000-10-01 23:35:00 18 2000-10-01 23:52:00 27 2000-10-02 00:09:00 39 2000-10-02 00:26:00 24 Freq: 17T, dtype: int64
>>> ts.resample('17min', origin='2000-01-01').sum() 2000-10-01 23:24:00 3 2000-10-01 23:41:00 15 2000-10-01 23:58:00 45 2000-10-02 00:15:00 45 Freq: 17T, dtype: int64
If you want to adjust the start of the bins with an offset Timedelta, the two following lines are equivalent:
>>> ts.resample('17min', origin='start').sum() 2000-10-01 23:30:00 9 2000-10-01 23:47:00 21 2000-10-02 00:04:00 54 2000-10-02 00:21:00 24 Freq: 17T, dtype: int64
>>> ts.resample('17min', offset='23h30min').sum() 2000-10-01 23:30:00 9 2000-10-01 23:47:00 21 2000-10-02 00:04:00 54 2000-10-02 00:21:00 24 Freq: 17T, dtype: int64
If you want to take the largest Timestamp as the end of the bins:
>>> ts.resample('17min', origin='end').sum() 2000-10-01 23:35:00 0 2000-10-01 23:52:00 18 2000-10-02 00:09:00 27 2000-10-02 00:26:00 63 Freq: 17T, dtype: int64
In contrast with the start_day, you can use end_day to take the ceiling midnight of the largest Timestamp as the end of the bins and drop the bins not containing data:
>>> ts.resample('17min', origin='end_day').sum() 2000-10-01 23:38:00 3 2000-10-01 23:55:00 15 2000-10-02 00:12:00 45 2000-10-02 00:29:00 45 Freq: 17T, dtype: int64
To replace the use of the deprecated base argument, you can now use offset, in this example it is equivalent to have base=2:
>>> ts.resample('17min', offset='2min').sum() 2000-10-01 23:16:00 0 2000-10-01 23:33:00 9 2000-10-01 23:50:00 36 2000-10-02 00:07:00 39 2000-10-02 00:24:00 24 Freq: 17T, dtype: int64
To replace the use of the deprecated loffset argument:
>>> from pandas.tseries.frequencies import to_offset >>> loffset = '19min' >>> ts_out = ts.resample('17min').sum() >>> ts_out.index = ts_out.index + to_offset(loffset) >>> ts_out 2000-10-01 23:33:00 0 2000-10-01 23:50:00 9 2000-10-02 00:07:00 21 2000-10-02 00:24:00 54 2000-10-02 00:41:00 24 Freq: 17T, dtype: int64
- reset_index(level: Hashable | Sequence[Hashable] | None = None, drop: bool = False, inplace: Literal[False] = False, col_level: Hashable = 0, col_fill: Hashable = '') DataFrame [source]¶
- reset_index(level: Hashable | Sequence[Hashable] | None, drop: bool, inplace: Literal[True], col_level: Hashable = 0, col_fill: Hashable = '') None
- reset_index(*, drop: bool, inplace: Literal[True], col_level: Hashable = 0, col_fill: Hashable = '') None
- reset_index(level: Hashable | Sequence[Hashable] | None, *, inplace: Literal[True], col_level: Hashable = 0, col_fill: Hashable = '') None
- reset_index(*, inplace: Literal[True], col_level: Hashable = 0, col_fill: Hashable = '') None
- reset_index(level: Hashable | Sequence[Hashable] | None = None, drop: bool = False, inplace: bool = False, col_level: Hashable = 0, col_fill: Hashable = '') DataFrame | None
Reset the index, or a level of it.
Reset the index of the DataFrame, and use the default one instead. If the DataFrame has a MultiIndex, this method can remove one or more levels.
- Parameters
level (int, str, tuple, or list, default None) – Only remove the given levels from the index. Removes all levels by default.
drop (bool, default False) – Do not try to insert index into dataframe columns. This resets the index to the default integer index.
inplace (bool, default False) – Modify the DataFrame in place (do not create a new object).
col_level (int or str, default 0) – If the columns have multiple levels, determines which level the labels are inserted into. By default it is inserted into the first level.
col_fill (object, default ‘’) – If the columns have multiple levels, determines how the other levels are named. If None then the index name is repeated.
- Returns
DataFrame with the new index or None if
inplace=True
.- Return type
DataFrame or None
See also
DataFrame.set_index
Opposite of reset_index.
DataFrame.reindex
Change to new indices or expand indices.
DataFrame.reindex_like
Change to same indices as other DataFrame.
Examples
>>> df = pd.DataFrame([('bird', 389.0), ... ('bird', 24.0), ... ('mammal', 80.5), ... ('mammal', np.nan)], ... index=['falcon', 'parrot', 'lion', 'monkey'], ... columns=('class', 'max_speed')) >>> df class max_speed falcon bird 389.0 parrot bird 24.0 lion mammal 80.5 monkey mammal NaN
When we reset the index, the old index is added as a column, and a new sequential index is used:
>>> df.reset_index() index class max_speed 0 falcon bird 389.0 1 parrot bird 24.0 2 lion mammal 80.5 3 monkey mammal NaN
We can use the drop parameter to avoid the old index being added as a column:
>>> df.reset_index(drop=True) class max_speed 0 bird 389.0 1 bird 24.0 2 mammal 80.5 3 mammal NaN
You can also use reset_index with MultiIndex.
>>> index = pd.MultiIndex.from_tuples([('bird', 'falcon'), ... ('bird', 'parrot'), ... ('mammal', 'lion'), ... ('mammal', 'monkey')], ... names=['class', 'name']) >>> columns = pd.MultiIndex.from_tuples([('speed', 'max'), ... ('species', 'type')]) >>> df = pd.DataFrame([(389.0, 'fly'), ... ( 24.0, 'fly'), ... ( 80.5, 'run'), ... (np.nan, 'jump')], ... index=index, ... columns=columns) >>> df speed species max type class name bird falcon 389.0 fly parrot 24.0 fly mammal lion 80.5 run monkey NaN jump
If the index has multiple levels, we can reset a subset of them:
>>> df.reset_index(level='class') class speed species max type name falcon bird 389.0 fly parrot bird 24.0 fly lion mammal 80.5 run monkey mammal NaN jump
If we are not dropping the index, by default, it is placed in the top level. We can place it in another level:
>>> df.reset_index(level='class', col_level=1) speed species class max type name falcon bird 389.0 fly parrot bird 24.0 fly lion mammal 80.5 run monkey mammal NaN jump
When the index is inserted under another level, we can specify under which one with the parameter col_fill:
>>> df.reset_index(level='class', col_level=1, col_fill='species') species speed species class max type name falcon bird 389.0 fly parrot bird 24.0 fly lion mammal 80.5 run monkey mammal NaN jump
If we specify a nonexistent level for col_fill, it is created:
>>> df.reset_index(level='class', col_level=1, col_fill='genus') genus speed species class max type name falcon bird 389.0 fly parrot bird 24.0 fly lion mammal 80.5 run monkey mammal NaN jump
- rfloordiv(other, axis='columns', level=None, fill_value=None)¶
Get Integer division of dataframe and other, element-wise (binary operator rfloordiv).
Equivalent to
other // dataframe
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, floordiv.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- rmod(other, axis='columns', level=None, fill_value=None)¶
Get Modulo of dataframe and other, element-wise (binary operator rmod).
Equivalent to
other % dataframe
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, mod.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- rmul(other, axis='columns', level=None, fill_value=None)¶
Get Multiplication of dataframe and other, element-wise (binary operator rmul).
Equivalent to
other * dataframe
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, mul.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- round(decimals=0, *args, **kwargs)[source]¶
Round a DataFrame to a variable number of decimal places.
- Parameters
decimals (int, dict, Series) – Number of decimal places to round each column to. If an int is given, round each column to the same number of places. Otherwise dict and Series round to variable numbers of places. Column names should be in the keys if decimals is a dict-like, or in the index if decimals is a Series. Any columns not included in decimals will be left as is. Elements of decimals which are not columns of the input will be ignored.
*args – Additional keywords have no effect but might be accepted for compatibility with numpy.
**kwargs – Additional keywords have no effect but might be accepted for compatibility with numpy.
- Returns
A DataFrame with the affected columns rounded to the specified number of decimal places.
- Return type
See also
numpy.around
Round a numpy array to the given number of decimals.
Series.round
Round a Series to the given number of decimals.
Examples
>>> df = pd.DataFrame([(.21, .32), (.01, .67), (.66, .03), (.21, .18)], ... columns=['dogs', 'cats']) >>> df dogs cats 0 0.21 0.32 1 0.01 0.67 2 0.66 0.03 3 0.21 0.18
By providing an integer each column is rounded to the same number of decimal places
>>> df.round(1) dogs cats 0 0.2 0.3 1 0.0 0.7 2 0.7 0.0 3 0.2 0.2
With a dict, the number of places for specific columns can be specified with the column names as key and the number of decimal places as value
>>> df.round({'dogs': 1, 'cats': 0}) dogs cats 0 0.2 0.0 1 0.0 1.0 2 0.7 0.0 3 0.2 0.0
Using a Series, the number of places for specific columns can be specified with the column names as index and the number of decimal places as value
>>> decimals = pd.Series([0, 1], index=['cats', 'dogs']) >>> df.round(decimals) dogs cats 0 0.2 0.0 1 0.0 1.0 2 0.7 0.0 3 0.2 0.0
- rpow(other, axis='columns', level=None, fill_value=None)¶
Get Exponential power of dataframe and other, element-wise (binary operator rpow).
Equivalent to
other ** dataframe
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, pow.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- rsub(other, axis='columns', level=None, fill_value=None)¶
Get Subtraction of dataframe and other, element-wise (binary operator rsub).
Equivalent to
other - dataframe
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, sub.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- rtruediv(other, axis='columns', level=None, fill_value=None)¶
Get Floating division of dataframe and other, element-wise (binary operator rtruediv).
Equivalent to
other / dataframe
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, truediv.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- select_dtypes(include=None, exclude=None)[source]¶
Return a subset of the DataFrame’s columns based on the column dtypes.
- Parameters
include, exclude (scalar or list-like) – A selection of dtypes or strings to be included/excluded. At least one of these parameters must be supplied.
- Returns
The subset of the frame including the dtypes in
include
and excluding the dtypes inexclude
.- Return type
- Raises
If both of
include
andexclude
are empty * Ifinclude
andexclude
have overlapping elements * If any kind of string dtype is passed in.
See also
DataFrame.dtypes
Return Series with the data type of each column.
Notes
To select all numeric types, use
np.number
or'number'
To select strings you must use the
object
dtype, but note that this will return all object dtype columnsSee the numpy dtype hierarchy
To select datetimes, use
np.datetime64
,'datetime'
or'datetime64'
To select timedeltas, use
np.timedelta64
,'timedelta'
or'timedelta64'
To select Pandas categorical dtypes, use
'category'
To select Pandas datetimetz dtypes, use
'datetimetz'
(new in 0.20.0) or'datetime64[ns, tz]'
Examples
>>> df = pd.DataFrame({'a': [1, 2] * 3, ... 'b': [True, False] * 3, ... 'c': [1.0, 2.0] * 3}) >>> df a b c 0 1 True 1.0 1 2 False 2.0 2 1 True 1.0 3 2 False 2.0 4 1 True 1.0 5 2 False 2.0
>>> df.select_dtypes(include='bool') b 0 True 1 False 2 True 3 False 4 True 5 False
>>> df.select_dtypes(include=['float64']) c 0 1.0 1 2.0 2 1.0 3 2.0 4 1.0 5 2.0
>>> df.select_dtypes(exclude=['int64']) b c 0 True 1.0 1 False 2.0 2 True 1.0 3 False 2.0 4 True 1.0 5 False 2.0
- sem(axis=None, skipna=True, level=None, ddof=1, numeric_only=None, **kwargs)¶
Return unbiased standard error of the mean over requested axis.
Normalized by N-1 by default. This can be changed using the ddof argument
- Parameters
axis ({index (0), columns (1)})
skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a Series.
ddof (int, default 1) – Delta Degrees of Freedom. The divisor used in calculations is N - ddof, where N represents the number of elements.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
- Return type
- set_axis(labels, axis: Axis = 0, inplace: Literal[False] = False) DataFrame [source]¶
- set_axis(labels, axis: Axis, inplace: Literal[True]) None
- set_axis(labels, *, inplace: Literal[True]) None
- set_axis(labels, axis: Axis = 0, inplace: bool = False) DataFrame | None
Assign desired index to given axis.
Indexes for column or row labels can be changed by assigning a list-like or Index.
- Parameters
labels (list-like, Index) – The values for the new index.
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – The axis to update. The value 0 identifies the rows, and 1 identifies the columns.
inplace (bool, default False) – Whether to return a new DataFrame instance.
- Returns
renamed – An object of type DataFrame or None if
inplace=True
.- Return type
DataFrame or None
See also
DataFrame.rename_axis
Alter the name of the index or columns. Examples ——– >>> df = pd.DataFrame({“A”: [1, 2, 3], “B”: [4, 5, 6]}) Change the row labels. >>> df.set_axis([‘a’, ‘b’, ‘c’], axis=’index’) A B a 1 4 b 2 5 c 3 6 Change the column labels. >>> df.set_axis([‘I’, ‘II’], axis=’columns’) I II 0 1 4 1 2 5 2 3 6 Now, update the labels inplace. >>> df.set_axis([‘i’, ‘ii’], axis=’columns’, inplace=True) >>> df i ii 0 1 4 1 2 5 2 3 6
- set_index(keys, drop=True, append=False, inplace=False, verify_integrity=False)[source]¶
Set the DataFrame index using existing columns.
Set the DataFrame index (row labels) using one or more existing columns or arrays (of the correct length). The index can replace the existing index or expand on it.
- Parameters
keys (label or array-like or list of labels/arrays) – This parameter can be either a single column key, a single array of the same length as the calling DataFrame, or a list containing an arbitrary combination of column keys and arrays. Here, “array” encompasses
Series
,Index
,np.ndarray
, and instances ofIterator
.drop (bool, default True) – Delete columns to be used as the new index.
append (bool, default False) – Whether to append columns to existing index.
inplace (bool, default False) – If True, modifies the DataFrame in place (do not create a new object).
verify_integrity (bool, default False) – Check the new index for duplicates. Otherwise defer the check until necessary. Setting to False will improve the performance of this method.
- Returns
Changed row labels or None if
inplace=True
.- Return type
DataFrame or None
See also
DataFrame.reset_index
Opposite of set_index.
DataFrame.reindex
Change to new indices or expand indices.
DataFrame.reindex_like
Change to same indices as other DataFrame.
Examples
>>> df = pd.DataFrame({'month': [1, 4, 7, 10], ... 'year': [2012, 2014, 2013, 2014], ... 'sale': [55, 40, 84, 31]}) >>> df month year sale 0 1 2012 55 1 4 2014 40 2 7 2013 84 3 10 2014 31
Set the index to become the ‘month’ column:
>>> df.set_index('month') year sale month 1 2012 55 4 2014 40 7 2013 84 10 2014 31
Create a MultiIndex using columns ‘year’ and ‘month’:
>>> df.set_index(['year', 'month']) sale year month 2012 1 55 2014 4 40 2013 7 84 2014 10 31
Create a MultiIndex using an Index and a column:
>>> df.set_index([pd.Index([1, 2, 3, 4]), 'year']) month sale year 1 2012 1 55 2 2014 4 40 3 2013 7 84 4 2014 10 31
Create a MultiIndex using two Series:
>>> s = pd.Series([1, 2, 3, 4]) >>> df.set_index([s, s**2]) month year sale 1 1 1 2012 55 2 4 4 2014 40 3 9 7 2013 84 4 16 10 2014 31
- property shape: tuple[int, int]¶
Return a tuple representing the dimensionality of the DataFrame.
See also
ndarray.shape
Tuple of array dimensions.
Examples
>>> df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]}) >>> df.shape (2, 2)
>>> df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4], ... 'col3': [5, 6]}) >>> df.shape (2, 3)
- shift(periods=1, freq=None, axis=0, fill_value=NoDefault.no_default)[source]¶
Shift index by desired number of periods with an optional time freq.
When freq is not passed, shift the index without realigning the data. If freq is passed (in this case, the index must be date or datetime, or it will raise a NotImplementedError), the index will be increased using the periods and the freq. freq can be inferred when specified as “infer” as long as either freq or inferred_freq attribute is set in the index.
- Parameters
periods (int) – Number of periods to shift. Can be positive or negative.
freq (DateOffset, tseries.offsets, timedelta, or str, optional) – Offset to use from the tseries module or time rule (e.g. ‘EOM’). If freq is specified then the index values are shifted but the data is not realigned. That is, use freq if you would like to extend the index when shifting and preserve the original data. If freq is specified as “infer” then it will be inferred from the freq or inferred_freq attributes of the index. If neither of those attributes exist, a ValueError is thrown.
axis ({0 or ‘index’, 1 or ‘columns’, None}, default None) – Shift direction.
fill_value (object, optional) – The scalar value to use for newly introduced missing values. the default depends on the dtype of self. For numeric data,
np.nan
is used. For datetime, timedelta, or period data, etc.NaT
is used. For extension dtypes,self.dtype.na_value
is used.Changed in version 1.1.0.
- Returns
Copy of input object, shifted.
- Return type
See also
Index.shift
Shift values of Index.
DatetimeIndex.shift
Shift values of DatetimeIndex.
PeriodIndex.shift
Shift values of PeriodIndex.
tshift
Shift the time index, using the index’s frequency if available.
Examples
>>> df = pd.DataFrame({"Col1": [10, 20, 15, 30, 45], ... "Col2": [13, 23, 18, 33, 48], ... "Col3": [17, 27, 22, 37, 52]}, ... index=pd.date_range("2020-01-01", "2020-01-05")) >>> df Col1 Col2 Col3 2020-01-01 10 13 17 2020-01-02 20 23 27 2020-01-03 15 18 22 2020-01-04 30 33 37 2020-01-05 45 48 52
>>> df.shift(periods=3) Col1 Col2 Col3 2020-01-01 NaN NaN NaN 2020-01-02 NaN NaN NaN 2020-01-03 NaN NaN NaN 2020-01-04 10.0 13.0 17.0 2020-01-05 20.0 23.0 27.0
>>> df.shift(periods=1, axis="columns") Col1 Col2 Col3 2020-01-01 NaN 10 13 2020-01-02 NaN 20 23 2020-01-03 NaN 15 18 2020-01-04 NaN 30 33 2020-01-05 NaN 45 48
>>> df.shift(periods=3, fill_value=0) Col1 Col2 Col3 2020-01-01 0 0 0 2020-01-02 0 0 0 2020-01-03 0 0 0 2020-01-04 10 13 17 2020-01-05 20 23 27
>>> df.shift(periods=3, freq="D") Col1 Col2 Col3 2020-01-04 10 13 17 2020-01-05 20 23 27 2020-01-06 15 18 22 2020-01-07 30 33 37 2020-01-08 45 48 52
>>> df.shift(periods=3, freq="infer") Col1 Col2 Col3 2020-01-04 10 13 17 2020-01-05 20 23 27 2020-01-06 15 18 22 2020-01-07 30 33 37 2020-01-08 45 48 52
- skew(axis: int | None | lib.NoDefault = NoDefault.no_default, skipna=True, level=None, numeric_only=None, **kwargs)¶
Return unbiased skew over requested axis.
Normalized by N-1.
- Parameters
axis ({index (0), columns (1)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a Series.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
**kwargs – Additional keyword arguments to be passed to the function.
- Return type
- sort_index(axis=0, level=None, ascending=True, inplace=False, kind='quicksort', na_position='last', sort_remaining=True, ignore_index=False, key=None)[source]¶
Sort object by labels (along an axis).
Returns a new DataFrame sorted by label if inplace argument is
False
, otherwise updates the original DataFrame and returns None.- Parameters
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – The axis along which to sort. The value 0 identifies the rows, and 1 identifies the columns.
level (int or level name or list of ints or list of level names) – If not None, sort on values in specified index level(s).
ascending (bool or list-like of bools, default True) – Sort ascending vs. descending. When the index is a MultiIndex the sort direction can be controlled for each level individually.
inplace (bool, default False) – If True, perform operation in-place.
kind ({‘quicksort’, ‘mergesort’, ‘heapsort’, ‘stable’}, default ‘quicksort’) – Choice of sorting algorithm. See also
numpy.sort()
for more information. mergesort and stable are the only stable algorithms. For DataFrames, this option is only applied when sorting on a single column or label.na_position ({‘first’, ‘last’}, default ‘last’) – Puts NaNs at the beginning if first; last puts NaNs at the end. Not implemented for MultiIndex.
sort_remaining (bool, default True) – If True and sorting by level and index is multilevel, sort by other levels too (in order) after sorting by specified level.
ignore_index (bool, default False) – If True, the resulting axis will be labeled 0, 1, …, n - 1.
New in version 1.0.0.
key (callable, optional) – If not None, apply the key function to the index values before sorting. This is similar to the key argument in the builtin
sorted()
function, with the notable difference that this key function should be vectorized. It should expect anIndex
and return anIndex
of the same shape. For MultiIndex inputs, the key is applied per level.New in version 1.1.0.
- Returns
The original DataFrame sorted by the labels or None if
inplace=True
.- Return type
DataFrame or None
See also
Series.sort_index
Sort Series by the index.
DataFrame.sort_values
Sort DataFrame by the value.
Series.sort_values
Sort Series by the value.
Examples
>>> df = pd.DataFrame([1, 2, 3, 4, 5], index=[100, 29, 234, 1, 150], ... columns=['A']) >>> df.sort_index() A 1 4 29 2 100 1 150 5 234 3
By default, it sorts in ascending order, to sort in descending order, use
ascending=False
>>> df.sort_index(ascending=False) A 234 3 150 5 100 1 29 2 1 4
A key function can be specified which is applied to the index before sorting. For a
MultiIndex
this is applied to each level separately.>>> df = pd.DataFrame({"a": [1, 2, 3, 4]}, index=['A', 'b', 'C', 'd']) >>> df.sort_index(key=lambda x: x.str.lower()) a A 1 b 2 C 3 d 4
- sort_values(by, axis=0, ascending=True, inplace=False, kind='quicksort', na_position='last', ignore_index=False, key=None)[source]¶
Sort by the values along either axis.
- Parameters
by (str or list of str) – Name or list of names to sort by.
if axis is 0 or ‘index’ then by may contain index levels and/or column labels.
if axis is 1 or ‘columns’ then by may contain column levels and/or index labels.
- axis{0 or ‘index’, 1 or ‘columns’}, default 0
Axis to be sorted.
- ascendingbool or list of bool, default True
Sort ascending vs. descending. Specify list for multiple sort orders. If this is a list of bools, must match the length of the by.
- inplacebool, default False
If True, perform operation in-place.
- kind{‘quicksort’, ‘mergesort’, ‘heapsort’, ‘stable’}, default ‘quicksort’
Choice of sorting algorithm. See also
numpy.sort()
for more information. mergesort and stable are the only stable algorithms. For DataFrames, this option is only applied when sorting on a single column or label.- na_position{‘first’, ‘last’}, default ‘last’
Puts NaNs at the beginning if first; last puts NaNs at the end.
- ignore_indexbool, default False
If True, the resulting axis will be labeled 0, 1, …, n - 1.
New in version 1.0.0.
- keycallable, optional
Apply the key function to the values before sorting. This is similar to the key argument in the builtin
sorted()
function, with the notable difference that this key function should be vectorized. It should expect aSeries
and return a Series with the same shape as the input. It will be applied to each column in by independently.New in version 1.1.0.
- Returns
DataFrame with sorted values or None if
inplace=True
.- Return type
DataFrame or None
See also
DataFrame.sort_index
Sort a DataFrame by the index.
Series.sort_values
Similar method for a Series.
Examples
>>> df = pd.DataFrame({ ... 'col1': ['A', 'A', 'B', np.nan, 'D', 'C'], ... 'col2': [2, 1, 9, 8, 7, 4], ... 'col3': [0, 1, 9, 4, 2, 3], ... 'col4': ['a', 'B', 'c', 'D', 'e', 'F'] ... }) >>> df col1 col2 col3 col4 0 A 2 0 a 1 A 1 1 B 2 B 9 9 c 3 NaN 8 4 D 4 D 7 2 e 5 C 4 3 F
Sort by col1
>>> df.sort_values(by=['col1']) col1 col2 col3 col4 0 A 2 0 a 1 A 1 1 B 2 B 9 9 c 5 C 4 3 F 4 D 7 2 e 3 NaN 8 4 D
Sort by multiple columns
>>> df.sort_values(by=['col1', 'col2']) col1 col2 col3 col4 1 A 1 1 B 0 A 2 0 a 2 B 9 9 c 5 C 4 3 F 4 D 7 2 e 3 NaN 8 4 D
Sort Descending
>>> df.sort_values(by='col1', ascending=False) col1 col2 col3 col4 4 D 7 2 e 5 C 4 3 F 2 B 9 9 c 0 A 2 0 a 1 A 1 1 B 3 NaN 8 4 D
Putting NAs first
>>> df.sort_values(by='col1', ascending=False, na_position='first') col1 col2 col3 col4 3 NaN 8 4 D 4 D 7 2 e 5 C 4 3 F 2 B 9 9 c 0 A 2 0 a 1 A 1 1 B
Sorting with a key function
>>> df.sort_values(by='col4', key=lambda col: col.str.lower()) col1 col2 col3 col4 0 A 2 0 a 1 A 1 1 B 2 B 9 9 c 3 NaN 8 4 D 4 D 7 2 e 5 C 4 3 F
Natural sort with the key argument, using the natsort <https://github.com/SethMMorton/natsort> package.
>>> df = pd.DataFrame({ ... "time": ['0hr', '128hr', '72hr', '48hr', '96hr'], ... "value": [10, 20, 30, 40, 50] ... }) >>> df time value 0 0hr 10 1 128hr 20 2 72hr 30 3 48hr 40 4 96hr 50 >>> from natsort import index_natsorted >>> df.sort_values( ... by="time", ... key=lambda x: np.argsort(index_natsorted(df["time"])) ... ) time value 0 0hr 10 3 48hr 40 2 72hr 30 4 96hr 50 1 128hr 20
- sparse¶
alias of
pandas.core.arrays.sparse.accessor.SparseFrameAccessor
- stack(level=- 1, dropna=True)[source]¶
Stack the prescribed level(s) from columns to index.
Return a reshaped DataFrame or Series having a multi-level index with one or more new inner-most levels compared to the current DataFrame. The new inner-most levels are created by pivoting the columns of the current dataframe:
if the columns have a single level, the output is a Series;
if the columns have multiple levels, the new index level(s) is (are) taken from the prescribed level(s) and the output is a DataFrame.
- Parameters
level (int, str, list, default -1) – Level(s) to stack from the column axis onto the index axis, defined as one index or label, or a list of indices or labels.
dropna (bool, default True) – Whether to drop rows in the resulting Frame/Series with missing values. Stacking a column level onto the index axis can create combinations of index and column values that are missing from the original dataframe. See Examples section.
- Returns
Stacked dataframe or series.
- Return type
See also
DataFrame.unstack
Unstack prescribed level(s) from index axis onto column axis.
DataFrame.pivot
Reshape dataframe from long format to wide format.
DataFrame.pivot_table
Create a spreadsheet-style pivot table as a DataFrame.
Notes
The function is named by analogy with a collection of books being reorganized from being side by side on a horizontal position (the columns of the dataframe) to being stacked vertically on top of each other (in the index of the dataframe).
Reference the user guide for more examples.
Examples
Single level columns
>>> df_single_level_cols = pd.DataFrame([[0, 1], [2, 3]], ... index=['cat', 'dog'], ... columns=['weight', 'height'])
Stacking a dataframe with a single level column axis returns a Series:
>>> df_single_level_cols weight height cat 0 1 dog 2 3 >>> df_single_level_cols.stack() cat weight 0 height 1 dog weight 2 height 3 dtype: int64
Multi level columns: simple case
>>> multicol1 = pd.MultiIndex.from_tuples([('weight', 'kg'), ... ('weight', 'pounds')]) >>> df_multi_level_cols1 = pd.DataFrame([[1, 2], [2, 4]], ... index=['cat', 'dog'], ... columns=multicol1)
Stacking a dataframe with a multi-level column axis:
>>> df_multi_level_cols1 weight kg pounds cat 1 2 dog 2 4 >>> df_multi_level_cols1.stack() weight cat kg 1 pounds 2 dog kg 2 pounds 4
Missing values
>>> multicol2 = pd.MultiIndex.from_tuples([('weight', 'kg'), ... ('height', 'm')]) >>> df_multi_level_cols2 = pd.DataFrame([[1.0, 2.0], [3.0, 4.0]], ... index=['cat', 'dog'], ... columns=multicol2)
It is common to have missing values when stacking a dataframe with multi-level columns, as the stacked dataframe typically has more values than the original dataframe. Missing values are filled with NaNs:
>>> df_multi_level_cols2 weight height kg m cat 1.0 2.0 dog 3.0 4.0 >>> df_multi_level_cols2.stack() height weight cat kg NaN 1.0 m 2.0 NaN dog kg NaN 3.0 m 4.0 NaN
Prescribing the level(s) to be stacked
The first parameter controls which level or levels are stacked:
>>> df_multi_level_cols2.stack(0) kg m cat height NaN 2.0 weight 1.0 NaN dog height NaN 4.0 weight 3.0 NaN >>> df_multi_level_cols2.stack([0, 1]) cat height m 2.0 weight kg 1.0 dog height m 4.0 weight kg 3.0 dtype: float64
Dropping missing values
>>> df_multi_level_cols3 = pd.DataFrame([[None, 1.0], [2.0, 3.0]], ... index=['cat', 'dog'], ... columns=multicol2)
Note that rows where all values are missing are dropped by default but this behaviour can be controlled via the dropna keyword parameter:
>>> df_multi_level_cols3 weight height kg m cat NaN 1.0 dog 2.0 3.0 >>> df_multi_level_cols3.stack(dropna=False) height weight cat kg NaN NaN m 1.0 NaN dog kg NaN 2.0 m 3.0 NaN >>> df_multi_level_cols3.stack(dropna=True) height weight cat m 1.0 NaN dog kg NaN 2.0 m 3.0 NaN
- std(axis=None, skipna=True, level=None, ddof=1, numeric_only=None, **kwargs)¶
Return sample standard deviation over requested axis.
Normalized by N-1 by default. This can be changed using the ddof argument.
- Parameters
axis ({index (0), columns (1)})
skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a Series.
ddof (int, default 1) – Delta Degrees of Freedom. The divisor used in calculations is N - ddof, where N represents the number of elements.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
- Return type
Notes
To have the same behaviour as numpy.std, use ddof=0 (instead of the default ddof=1)
Examples
>>> df = pd.DataFrame({'person_id': [0, 1, 2, 3], ... 'age': [21, 25, 62, 43], ... 'height': [1.61, 1.87, 1.49, 2.01]} ... ).set_index('person_id') >>> df age height person_id 0 21 1.61 1 25 1.87 2 62 1.49 3 43 2.01
The standard deviation of the columns can be found as follows:
>>> df.std() age 18.786076 height 0.237417
Alternatively, ddof=0 can be set to normalize by N instead of N-1:
>>> df.std(ddof=0) age 16.269219 height 0.205609
- property style: pandas.io.formats.style.Styler¶
Returns a Styler object.
Contains methods for building a styled HTML representation of the DataFrame.
See also
io.formats.style.Styler
Helps style a DataFrame or Series according to the data with HTML and CSS.
- Return type
- sub(other, axis='columns', level=None, fill_value=None)¶
Get Subtraction of dataframe and other, element-wise (binary operator sub).
Equivalent to
dataframe - other
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, rsub.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- subtract(other, axis='columns', level=None, fill_value=None)¶
Get Subtraction of dataframe and other, element-wise (binary operator sub).
Equivalent to
dataframe - other
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, rsub.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- sum(axis=None, skipna=True, level=None, numeric_only=None, min_count=0, **kwargs)¶
Return the sum of the values over the requested axis.
This is equivalent to the method
numpy.sum
.- Parameters
axis ({index (0), columns (1)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a Series.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
min_count (int, default 0) – The required number of valid values to perform the operation. If fewer than
min_count
non-NA values are present the result will be NA.**kwargs – Additional keyword arguments to be passed to the function.
- Return type
See also
Series.sum
Return the sum.
Series.min
Return the minimum.
Series.max
Return the maximum.
Series.idxmin
Return the index of the minimum.
Series.idxmax
Return the index of the maximum.
DataFrame.sum
Return the sum over the requested axis.
DataFrame.min
Return the minimum over the requested axis.
DataFrame.max
Return the maximum over the requested axis.
DataFrame.idxmin
Return the index of the minimum over the requested axis.
DataFrame.idxmax
Return the index of the maximum over the requested axis.
Examples
>>> idx = pd.MultiIndex.from_arrays([ ... ['warm', 'warm', 'cold', 'cold'], ... ['dog', 'falcon', 'fish', 'spider']], ... names=['blooded', 'animal']) >>> s = pd.Series([4, 2, 0, 8], name='legs', index=idx) >>> s blooded animal warm dog 4 falcon 2 cold fish 0 spider 8 Name: legs, dtype: int64
>>> s.sum() 14
By default, the sum of an empty or all-NA Series is
0
.>>> pd.Series([], dtype="float64").sum() # min_count=0 is the default 0.0
This can be controlled with the
min_count
parameter. For example, if you’d like the sum of an empty series to be NaN, passmin_count=1
.>>> pd.Series([], dtype="float64").sum(min_count=1) nan
Thanks to the
skipna
parameter,min_count
handles all-NA and empty series identically.>>> pd.Series([np.nan]).sum() 0.0
>>> pd.Series([np.nan]).sum(min_count=1) nan
- swaplevel(i=- 2, j=- 1, axis=0)[source]¶
Swap levels i and j in a
MultiIndex
.Default is to swap the two innermost levels of the index.
- Parameters
i, j (int or str) – Levels of the indices to be swapped. Can pass level name as string.
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – The axis to swap levels on. 0 or ‘index’ for row-wise, 1 or ‘columns’ for column-wise.
- Returns
DataFrame with levels swapped in MultiIndex.
- Return type
Examples
>>> df = pd.DataFrame( ... {"Grade": ["A", "B", "A", "C"]}, ... index=[ ... ["Final exam", "Final exam", "Coursework", "Coursework"], ... ["History", "Geography", "History", "Geography"], ... ["January", "February", "March", "April"], ... ], ... ) >>> df Grade Final exam History January A Geography February B Coursework History March A Geography April C
In the following example, we will swap the levels of the indices. Here, we will swap the levels column-wise, but levels can be swapped row-wise in a similar manner. Note that column-wise is the default behaviour. By not supplying any arguments for i and j, we swap the last and second to last indices.
>>> df.swaplevel() Grade Final exam January History A February Geography B Coursework March History A April Geography C
By supplying one argument, we can choose which index to swap the last index with. We can for example swap the first index with the last one as follows.
>>> df.swaplevel(0) Grade January History Final exam A February Geography Final exam B March History Coursework A April Geography Coursework C
We can also define explicitly which indices we want to swap by supplying values for both i and j. Here, we for example swap the first and second indices.
>>> df.swaplevel(0, 1) Grade History Final exam January A Geography Final exam February B History Coursework March A Geography Coursework April C
- to_dict(orient='dict', into=<class 'dict'>)[source]¶
Convert the DataFrame to a dictionary.
The type of the key-value pairs can be customized with the parameters (see below).
- Parameters
orient (str {‘dict’, ‘list’, ‘series’, ‘split’, ‘records’, ‘index’}) – Determines the type of the values of the dictionary.
‘dict’ (default) : dict like {column -> {index -> value}}
‘list’ : dict like {column -> [values]}
‘series’ : dict like {column -> Series(values)}
‘split’ : dict like {‘index’ -> [index], ‘columns’ -> [columns], ‘data’ -> [values]}
‘tight’ : dict like {‘index’ -> [index], ‘columns’ -> [columns], ‘data’ -> [values], ‘index_names’ -> [index.names], ‘column_names’ -> [column.names]}
‘records’ : list like [{column -> value}, … , {column -> value}]
‘index’ : dict like {index -> {column -> value}}
Abbreviations are allowed. s indicates series and sp indicates split.
New in version 1.4.0: ‘tight’ as an allowed value for the
orient
argumentinto (class, default dict) – The collections.abc.Mapping subclass used for all Mappings in the return value. Can be the actual class or an empty instance of the mapping type you want. If you want a collections.defaultdict, you must pass it initialized.
- Returns
Return a collections.abc.Mapping object representing the DataFrame. The resulting transformation depends on the orient parameter.
- Return type
See also
DataFrame.from_dict
Create a DataFrame from a dictionary.
DataFrame.to_json
Convert a DataFrame to JSON format.
Examples
>>> df = pd.DataFrame({'col1': [1, 2], ... 'col2': [0.5, 0.75]}, ... index=['row1', 'row2']) >>> df col1 col2 row1 1 0.50 row2 2 0.75 >>> df.to_dict() {'col1': {'row1': 1, 'row2': 2}, 'col2': {'row1': 0.5, 'row2': 0.75}}
You can specify the return orientation.
>>> df.to_dict('series') {'col1': row1 1 row2 2 Name: col1, dtype: int64, 'col2': row1 0.50 row2 0.75 Name: col2, dtype: float64}
>>> df.to_dict('split') {'index': ['row1', 'row2'], 'columns': ['col1', 'col2'], 'data': [[1, 0.5], [2, 0.75]]}
>>> df.to_dict('records') [{'col1': 1, 'col2': 0.5}, {'col1': 2, 'col2': 0.75}]
>>> df.to_dict('index') {'row1': {'col1': 1, 'col2': 0.5}, 'row2': {'col1': 2, 'col2': 0.75}}
>>> df.to_dict('tight') {'index': ['row1', 'row2'], 'columns': ['col1', 'col2'], 'data': [[1, 0.5], [2, 0.75]], 'index_names': [None], 'column_names': [None]}
You can also specify the mapping type.
>>> from collections import OrderedDict, defaultdict >>> df.to_dict(into=OrderedDict) OrderedDict([('col1', OrderedDict([('row1', 1), ('row2', 2)])), ('col2', OrderedDict([('row1', 0.5), ('row2', 0.75)]))])
If you want a defaultdict, you need to initialize it:
>>> dd = defaultdict(list) >>> df.to_dict('records', into=dd) [defaultdict(<class 'list'>, {'col1': 1, 'col2': 0.5}), defaultdict(<class 'list'>, {'col1': 2, 'col2': 0.75})]
- to_feather(path, **kwargs)[source]¶
Write a DataFrame to the binary Feather format.
- Parameters
path (str, path object, file-like object) – String, path object (implementing
os.PathLike[str]
), or file-like object implementing a binarywrite()
function. If a string or a path, it will be used as Root Directory path when writing a partitioned dataset.**kwargs – Additional keywords passed to
pyarrow.feather.write_feather()
. Starting with pyarrow 0.17, this includes the compression, compression_level, chunksize and version keywords.New in version 1.1.0.
Notes
This function writes the dataframe as a feather file. Requires a default index. For saving the DataFrame with your custom index use a method that supports custom indices e.g. to_parquet.
- Return type
None
- to_gbq(destination_table, project_id=None, chunksize=None, reauth=False, if_exists='fail', auth_local_webserver=False, table_schema=None, location=None, progress_bar=True, credentials=None)[source]¶
Write a DataFrame to a Google BigQuery table.
This function requires the pandas-gbq package.
See the How to authenticate with Google BigQuery guide for authentication instructions.
- Parameters
destination_table (str) – Name of table to be written, in the form
dataset.tablename
.project_id (str, optional) – Google BigQuery Account project ID. Optional when available from the environment.
chunksize (int, optional) – Number of rows to be inserted in each chunk from the dataframe. Set to
None
to load the whole dataframe at once.reauth (bool, default False) – Force Google BigQuery to re-authenticate the user. This is useful if multiple accounts are used.
if_exists (str, default ‘fail’) – Behavior when the destination table exists. Value can be one of:
'fail'
If table exists raise pandas_gbq.gbq.TableCreationError.
'replace'
If table exists, drop it, recreate it, and insert data.
'append'
If table exists, insert data. Create if does not exist.
auth_local_webserver (bool, default False) – Use the `local webserver flow`_ instead of the `console flow`_ when getting user credentials.
New in version 0.2.0 of pandas-gbq.
table_schema (list of dicts, optional) – List of BigQuery table fields to which according DataFrame columns conform to, e.g.
[{'name': 'col1', 'type': 'STRING'},...]
. If schema is not provided, it will be generated according to dtypes of DataFrame columns. See BigQuery API documentation on available names of a field.New in version 0.3.1 of pandas-gbq.
location (str, optional) – Location where the load job should run. See the BigQuery locations documentation for a list of available locations. The location must match that of the target dataset.
New in version 0.5.0 of pandas-gbq.
progress_bar (bool, default True) – Use the library tqdm to show the progress bar for the upload, chunk by chunk.
New in version 0.5.0 of pandas-gbq.
credentials (google.auth.credentials.Credentials, optional) – Credentials for accessing Google APIs. Use this parameter to override default credentials, such as to use Compute Engine
google.auth.compute_engine.Credentials
or Service Accountgoogle.oauth2.service_account.Credentials
directly.New in version 0.8.0 of pandas-gbq.
See also
pandas_gbq.to_gbq
This function in the pandas-gbq library.
read_gbq
Read a DataFrame from Google BigQuery.
- Return type
None
- to_html(buf=None, columns=None, col_space=None, header=True, index=True, na_rep='NaN', formatters=None, float_format=None, sparsify=None, index_names=True, justify=None, max_rows=None, max_cols=None, show_dimensions=False, decimal='.', bold_rows=True, classes=None, escape=True, notebook=False, border=None, table_id=None, render_links=False, encoding=None)[source]¶
Render a DataFrame as an HTML table.
- Parameters
buf (str, Path or StringIO-like, optional, default None) – Buffer to write to. If None, the output is returned as a string.
columns (sequence, optional, default None) – The subset of columns to write. Writes all columns by default.
col_space (str or int, list or dict of int or str, optional) – The minimum width of each column in CSS length units. An int is assumed to be px units.
New in version 0.25.0: Ability to use str.
header (bool, optional) – Whether to print column labels, default True.
index (bool, optional, default True) – Whether to print index (row) labels.
na_rep (str, optional, default ‘NaN’) – String representation of
NaN
to use.formatters (list, tuple or dict of one-param. functions, optional) – Formatter functions to apply to columns’ elements by position or name. The result of each function must be a unicode string. List/tuple must be of length equal to the number of columns.
float_format (one-parameter function, optional, default None) – Formatter function to apply to columns’ elements if they are floats. This function must return a unicode string and will be applied only to the non-
NaN
elements, withNaN
being handled byna_rep
.Changed in version 1.2.0.
sparsify (bool, optional, default True) – Set to False for a DataFrame with a hierarchical index to print every multiindex key at each row.
index_names (bool, optional, default True) – Prints the names of the indexes.
justify (str, default None) – How to justify the column labels. If None uses the option from the print configuration (controlled by set_option), ‘right’ out of the box. Valid values are
left
right
center
justify
justify-all
start
end
inherit
match-parent
initial
unset.
max_rows (int, optional) – Maximum number of rows to display in the console.
max_cols (int, optional) – Maximum number of columns to display in the console.
show_dimensions (bool, default False) – Display DataFrame dimensions (number of rows by number of columns).
decimal (str, default ‘.’) – Character recognized as decimal separator, e.g. ‘,’ in Europe.
bold_rows (bool, default True) – Make the row labels bold in the output.
classes (str or list or tuple, default None) – CSS class(es) to apply to the resulting html table.
escape (bool, default True) – Convert the characters <, >, and & to HTML-safe sequences.
notebook ({True, False}, default False) – Whether the generated HTML is for IPython Notebook.
border (int) – A
border=border
attribute is included in the opening <table> tag. Defaultpd.options.display.html.border
.table_id (str, optional) – A css id is included in the opening <table> tag if specified.
render_links (bool, default False) – Convert URLs to HTML links.
encoding (str, default “utf-8”) – Set character encoding.
New in version 1.0.
- Returns
If buf is None, returns the result as a string. Otherwise returns None.
- Return type
str or None
See also
to_string
Convert DataFrame to a string.
- to_markdown(buf=None, mode='wt', index=True, storage_options=None, **kwargs)[source]¶
Print DataFrame in Markdown-friendly format.
New in version 1.0.0.
- Parameters
buf (str, Path or StringIO-like, optional, default None) – Buffer to write to. If None, the output is returned as a string.
mode (str, optional) – Mode in which file is opened, “wt” by default.
index (bool, optional, default True) – Add index (row) labels.
New in version 1.1.0.
storage_options (dict, optional) – Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc. For HTTP(S) URLs the key-value pairs are forwarded to
urllib
as header options. For other URLs (e.g. starting with “s3://”, and “gcs://”) the key-value pairs are forwarded tofsspec
. Please seefsspec
andurllib
for more details.New in version 1.2.0.
**kwargs – These parameters will be passed to tabulate.
- Returns
DataFrame in Markdown-friendly format.
- Return type
Notes
Requires the tabulate package.
- Examples
>>> df = pd.DataFrame( ... data={"animal_1": ["elk", "pig"], "animal_2": ["dog", "quetzal"]} ... ) >>> print(df.to_markdown()) | | animal_1 | animal_2 | |---:|:-----------|:-----------| | 0 | elk | dog | | 1 | pig | quetzal |
Output markdown with a tabulate option.
>>> print(df.to_markdown(tablefmt="grid")) +----+------------+------------+ | | animal_1 | animal_2 | +====+============+============+ | 0 | elk | dog | +----+------------+------------+ | 1 | pig | quetzal | +----+------------+------------+
- to_numpy(dtype=None, copy=False, na_value=NoDefault.no_default)[source]¶
Convert the DataFrame to a NumPy array.
By default, the dtype of the returned array will be the common NumPy dtype of all types in the DataFrame. For example, if the dtypes are
float16
andfloat32
, the results dtype will befloat32
. This may require copying data and coercing values, which may be expensive.- Parameters
dtype (str or numpy.dtype, optional) – The dtype to pass to
numpy.asarray()
.copy (bool, default False) – Whether to ensure that the returned value is not a view on another array. Note that
copy=False
does not ensure thatto_numpy()
is no-copy. Rather,copy=True
ensure that a copy is made, even if not strictly necessary.na_value (Any, optional) – The value to use for missing values. The default value depends on dtype and the dtypes of the DataFrame columns.
New in version 1.1.0.
- Return type
See also
Series.to_numpy
Similar method for Series.
Examples
>>> pd.DataFrame({"A": [1, 2], "B": [3, 4]}).to_numpy() array([[1, 3], [2, 4]])
With heterogeneous data, the lowest common type will have to be used.
>>> df = pd.DataFrame({"A": [1, 2], "B": [3.0, 4.5]}) >>> df.to_numpy() array([[1. , 3. ], [2. , 4.5]])
For a mix of numeric and non-numeric types, the output array will have object dtype.
>>> df['C'] = pd.date_range('2000', periods=2) >>> df.to_numpy() array([[1, 3.0, Timestamp('2000-01-01 00:00:00')], [2, 4.5, Timestamp('2000-01-02 00:00:00')]], dtype=object)
- to_parquet(path=None, engine='auto', compression='snappy', index=None, partition_cols=None, storage_options=None, **kwargs)[source]¶
Write a DataFrame to the binary parquet format.
This function writes the dataframe as a parquet file. You can choose different parquet backends, and have the option of compression. See the user guide for more details.
- Parameters
path (str, path object, file-like object, or None, default None) – String, path object (implementing
os.PathLike[str]
), or file-like object implementing a binarywrite()
function. If None, the result is returned as bytes. If a string or path, it will be used as Root Directory path when writing a partitioned dataset.Changed in version 1.2.0.
Previously this was “fname”
engine ({‘auto’, ‘pyarrow’, ‘fastparquet’}, default ‘auto’) – Parquet library to use. If ‘auto’, then the option
io.parquet.engine
is used. The defaultio.parquet.engine
behavior is to try ‘pyarrow’, falling back to ‘fastparquet’ if ‘pyarrow’ is unavailable.compression ({‘snappy’, ‘gzip’, ‘brotli’, None}, default ‘snappy’) – Name of the compression to use. Use
None
for no compression.index (bool, default None) – If
True
, include the dataframe’s index(es) in the file output. IfFalse
, they will not be written to the file. IfNone
, similar toTrue
the dataframe’s index(es) will be saved. However, instead of being saved as values, the RangeIndex will be stored as a range in the metadata so it doesn’t require much space and is faster. Other indexes will be included as columns in the file output.partition_cols (list, optional, default None) – Column names by which to partition the dataset. Columns are partitioned in the order they are given. Must be None if path is not a string.
storage_options (dict, optional) – Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc. For HTTP(S) URLs the key-value pairs are forwarded to
urllib
as header options. For other URLs (e.g. starting with “s3://”, and “gcs://”) the key-value pairs are forwarded tofsspec
. Please seefsspec
andurllib
for more details.New in version 1.2.0.
**kwargs – Additional arguments passed to the parquet library. See pandas io for more details.
- Return type
bytes if no path argument is provided else None
See also
read_parquet
Read a parquet file.
DataFrame.to_csv
Write a csv file.
DataFrame.to_sql
Write to a sql table.
DataFrame.to_hdf
Write to hdf.
Notes
This function requires either the fastparquet or pyarrow library.
Examples
>>> df = pd.DataFrame(data={'col1': [1, 2], 'col2': [3, 4]}) >>> df.to_parquet('df.parquet.gzip', ... compression='gzip') >>> pd.read_parquet('df.parquet.gzip') col1 col2 0 1 3 1 2 4
If you want to get a buffer to the parquet content you can use a io.BytesIO object, as long as you don’t use partition_cols, which creates multiple files.
>>> import io >>> f = io.BytesIO() >>> df.to_parquet(f) >>> f.seek(0) 0 >>> content = f.read()
- to_period(freq=None, axis=0, copy=True)[source]¶
Convert DataFrame from DatetimeIndex to PeriodIndex.
Convert DataFrame from DatetimeIndex to PeriodIndex with desired frequency (inferred from index if not passed).
- Parameters
freq (str, default) – Frequency of the PeriodIndex.
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – The axis to convert (the index by default).
copy (bool, default True) – If False then underlying input data is not copied.
- Return type
DataFrame with PeriodIndex
Examples
>>> idx = pd.to_datetime( ... [ ... "2001-03-31 00:00:00", ... "2002-05-31 00:00:00", ... "2003-08-31 00:00:00", ... ] ... )
>>> idx DatetimeIndex(['2001-03-31', '2002-05-31', '2003-08-31'], dtype='datetime64[ns]', freq=None)
>>> idx.to_period("M") PeriodIndex(['2001-03', '2002-05', '2003-08'], dtype='period[M]')
For the yearly frequency
>>> idx.to_period("Y") PeriodIndex(['2001', '2002', '2003'], dtype='period[A-DEC]')
- to_records(index=True, column_dtypes=None, index_dtypes=None)[source]¶
Convert DataFrame to a NumPy record array.
Index will be included as the first field of the record array if requested.
- Parameters
index (bool, default True) – Include index in resulting record array, stored in ‘index’ field or using the index label, if set.
column_dtypes (str, type, dict, default None) – If a string or type, the data type to store all columns. If a dictionary, a mapping of column names and indices (zero-indexed) to specific data types.
index_dtypes (str, type, dict, default None) – If a string or type, the data type to store all index levels. If a dictionary, a mapping of index level names and indices (zero-indexed) to specific data types.
This mapping is applied only if index=True.
- Returns
NumPy ndarray with the DataFrame labels as fields and each row of the DataFrame as entries.
- Return type
See also
DataFrame.from_records
Convert structured or record ndarray to DataFrame.
numpy.recarray
An ndarray that allows field access using attributes, analogous to typed columns in a spreadsheet.
Examples
>>> df = pd.DataFrame({'A': [1, 2], 'B': [0.5, 0.75]}, ... index=['a', 'b']) >>> df A B a 1 0.50 b 2 0.75 >>> df.to_records() rec.array([('a', 1, 0.5 ), ('b', 2, 0.75)], dtype=[('index', 'O'), ('A', '<i8'), ('B', '<f8')])
If the DataFrame index has no label then the recarray field name is set to ‘index’. If the index has a label then this is used as the field name:
>>> df.index = df.index.rename("I") >>> df.to_records() rec.array([('a', 1, 0.5 ), ('b', 2, 0.75)], dtype=[('I', 'O'), ('A', '<i8'), ('B', '<f8')])
The index can be excluded from the record array:
>>> df.to_records(index=False) rec.array([(1, 0.5 ), (2, 0.75)], dtype=[('A', '<i8'), ('B', '<f8')])
Data types can be specified for the columns:
>>> df.to_records(column_dtypes={"A": "int32"}) rec.array([('a', 1, 0.5 ), ('b', 2, 0.75)], dtype=[('I', 'O'), ('A', '<i4'), ('B', '<f8')])
As well as for the index:
>>> df.to_records(index_dtypes="<S2") rec.array([(b'a', 1, 0.5 ), (b'b', 2, 0.75)], dtype=[('I', 'S2'), ('A', '<i8'), ('B', '<f8')])
>>> index_dtypes = f"<S{df.index.str.len().max()}" >>> df.to_records(index_dtypes=index_dtypes) rec.array([(b'a', 1, 0.5 ), (b'b', 2, 0.75)], dtype=[('I', 'S1'), ('A', '<i8'), ('B', '<f8')])
- to_stata(path, convert_dates=None, write_index=True, byteorder=None, time_stamp=None, data_label=None, variable_labels=None, version=114, convert_strl=None, compression='infer', storage_options=None, *, value_labels=None)[source]¶
Export DataFrame object to Stata dta format.
Writes the DataFrame to a Stata dataset file. “dta” files contain a Stata dataset.
- Parameters
path (str, path object, or buffer) – String, path object (implementing
os.PathLike[str]
), or file-like object implementing a binarywrite()
function.Changed in version 1.0.0.
Previously this was “fname”
convert_dates (dict) – Dictionary mapping columns containing datetime types to stata internal format to use when writing the dates. Options are ‘tc’, ‘td’, ‘tm’, ‘tw’, ‘th’, ‘tq’, ‘ty’. Column can be either an integer or a name. Datetime columns that do not have a conversion type specified will be converted to ‘tc’. Raises NotImplementedError if a datetime column has timezone information.
write_index (bool) – Write the index to Stata dataset.
byteorder (str) – Can be “>”, “<”, “little”, or “big”. default is sys.byteorder.
time_stamp (datetime) – A datetime to use as file creation date. Default is the current time.
data_label (str, optional) – A label for the data set. Must be 80 characters or smaller.
variable_labels (dict) – Dictionary containing columns as keys and variable labels as values. Each label must be 80 characters or smaller.
version ({114, 117, 118, 119, None}, default 114) – Version to use in the output dta file. Set to None to let pandas decide between 118 or 119 formats depending on the number of columns in the frame. Version 114 can be read by Stata 10 and later. Version 117 can be read by Stata 13 or later. Version 118 is supported in Stata 14 and later. Version 119 is supported in Stata 15 and later. Version 114 limits string variables to 244 characters or fewer while versions 117 and later allow strings with lengths up to 2,000,000 characters. Versions 118 and 119 support Unicode characters, and version 119 supports more than 32,767 variables.
Version 119 should usually only be used when the number of variables exceeds the capacity of dta format 118. Exporting smaller datasets in format 119 may have unintended consequences, and, as of November 2020, Stata SE cannot read version 119 files.
Changed in version 1.0.0: Added support for formats 118 and 119.
convert_strl (list, optional) – List of column names to convert to string columns to Stata StrL format. Only available if version is 117. Storing strings in the StrL format can produce smaller dta files if strings have more than 8 characters and values are repeated.
compression (str or dict, default ‘infer’) – For on-the-fly compression of the output data. If ‘infer’ and ‘path’ path-like, then detect compression from the following extensions: ‘.gz’, ‘.bz2’, ‘.zip’, ‘.xz’, or ‘.zst’ (otherwise no compression). Set to
None
for no compression. Can also be a dict with key'method'
set to one of {'zip'
,'gzip'
,'bz2'
,'zstd'
} and other key-value pairs are forwarded tozipfile.ZipFile
,gzip.GzipFile
,bz2.BZ2File
, orzstandard.ZstdDecompressor
, respectively. As an example, the following could be passed for faster compression and to create a reproducible gzip archive:compression={'method': 'gzip', 'compresslevel': 1, 'mtime': 1}
.New in version 1.1.0.
Changed in version 1.4.0: Zstandard support.
storage_options (dict, optional) – Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc. For HTTP(S) URLs the key-value pairs are forwarded to
urllib
as header options. For other URLs (e.g. starting with “s3://”, and “gcs://”) the key-value pairs are forwarded tofsspec
. Please seefsspec
andurllib
for more details.New in version 1.2.0.
value_labels (dict of dicts) – Dictionary containing columns as keys and dictionaries of column value to labels as values. Labels for a single variable must be 32,000 characters or smaller.
New in version 1.4.0.
- Raises
If datetimes contain timezone information * Column dtype is not representable in Stata
Columns listed in convert_dates are neither datetime64[ns] or datetime.datetime * Column listed in convert_dates is not in DataFrame * Categorical label contains more than 32,000 characters
See also
read_stata
Import Stata data files.
io.stata.StataWriter
Low-level writer for Stata data files.
io.stata.StataWriter117
Low-level writer for version 117 files.
Examples
>>> df = pd.DataFrame({'animal': ['falcon', 'parrot', 'falcon', ... 'parrot'], ... 'speed': [350, 18, 361, 15]}) >>> df.to_stata('animals.dta')
- Return type
None
- to_string(buf: None = None, columns: Sequence[str] | None = None, col_space: int | list[int] | dict[Hashable, int] | None = None, header: bool | Sequence[str] = True, index: bool = True, na_rep: str = 'NaN', formatters: fmt.FormattersType | None = None, float_format: fmt.FloatFormatType | None = None, sparsify: bool | None = None, index_names: bool = True, justify: str | None = None, max_rows: int | None = None, max_cols: int | None = None, show_dimensions: bool = False, decimal: str = '.', line_width: int | None = None, min_rows: int | None = None, max_colwidth: int | None = None, encoding: str | None = None) str [source]¶
- to_string(buf: FilePath | WriteBuffer[str], columns: Sequence[str] | None = None, col_space: int | list[int] | dict[Hashable, int] | None = None, header: bool | Sequence[str] = True, index: bool = True, na_rep: str = 'NaN', formatters: fmt.FormattersType | None = None, float_format: fmt.FloatFormatType | None = None, sparsify: bool | None = None, index_names: bool = True, justify: str | None = None, max_rows: int | None = None, max_cols: int | None = None, show_dimensions: bool = False, decimal: str = '.', line_width: int | None = None, min_rows: int | None = None, max_colwidth: int | None = None, encoding: str | None = None) None
Render a DataFrame to a console-friendly tabular output.
- Parameters
buf (str, Path or StringIO-like, optional, default None) – Buffer to write to. If None, the output is returned as a string.
columns (sequence, optional, default None) – The subset of columns to write. Writes all columns by default.
col_space (int, list or dict of int, optional) – The minimum width of each column. If a list of ints is given every integers corresponds with one column. If a dict is given, the key references the column, while the value defines the space to use..
header (bool or sequence of str, optional) – Write out the column names. If a list of strings is given, it is assumed to be aliases for the column names.
index (bool, optional, default True) – Whether to print index (row) labels.
na_rep (str, optional, default ‘NaN’) – String representation of
NaN
to use.formatters (list, tuple or dict of one-param. functions, optional) – Formatter functions to apply to columns’ elements by position or name. The result of each function must be a unicode string. List/tuple must be of length equal to the number of columns.
float_format (one-parameter function, optional, default None) – Formatter function to apply to columns’ elements if they are floats. This function must return a unicode string and will be applied only to the non-
NaN
elements, withNaN
being handled byna_rep
.Changed in version 1.2.0.
sparsify (bool, optional, default True) – Set to False for a DataFrame with a hierarchical index to print every multiindex key at each row.
index_names (bool, optional, default True) – Prints the names of the indexes.
justify (str, default None) – How to justify the column labels. If None uses the option from the print configuration (controlled by set_option), ‘right’ out of the box. Valid values are
left
right
center
justify
justify-all
start
end
inherit
match-parent
initial
unset.
max_rows (int, optional) – Maximum number of rows to display in the console.
max_cols (int, optional) – Maximum number of columns to display in the console.
show_dimensions (bool, default False) – Display DataFrame dimensions (number of rows by number of columns).
decimal (str, default ‘.’) – Character recognized as decimal separator, e.g. ‘,’ in Europe.
line_width (int, optional) – Width to wrap a line in characters.
min_rows (int, optional) – The number of rows to display in the console in a truncated repr (when number of rows is above max_rows).
max_colwidth (int, optional) – Max width to truncate each column in characters. By default, no limit.
New in version 1.0.0.
encoding (str, default “utf-8”) – Set character encoding.
New in version 1.0.
- Returns
If buf is None, returns the result as a string. Otherwise returns None.
- Return type
str or None
See also
to_html
Convert DataFrame to HTML.
Examples
>>> d = {'col1': [1, 2, 3], 'col2': [4, 5, 6]} >>> df = pd.DataFrame(d) >>> print(df.to_string()) col1 col2 0 1 4 1 2 5 2 3 6
- to_timestamp(freq=None, how='start', axis=0, copy=True)[source]¶
Cast to DatetimeIndex of timestamps, at beginning of period.
- Parameters
freq (str, default frequency of PeriodIndex) – Desired frequency.
how ({‘s’, ‘e’, ‘start’, ‘end’}) – Convention for converting period to timestamp; start of period vs. end.
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – The axis to convert (the index by default).
copy (bool, default True) – If False then underlying input data is not copied.
- Return type
DataFrame with DatetimeIndex
- to_xml(path_or_buffer=None, index=True, root_name='data', row_name='row', na_rep=None, attr_cols=None, elem_cols=None, namespaces=None, prefix=None, encoding='utf-8', xml_declaration=True, pretty_print=True, parser='lxml', stylesheet=None, compression='infer', storage_options=None)[source]¶
Render a DataFrame to an XML document.
New in version 1.3.0.
- Parameters
path_or_buffer (str, path object, file-like object, or None, default None) – String, path object (implementing
os.PathLike[str]
), or file-like object implementing awrite()
function. If None, the result is returned as a string.index (bool, default True) – Whether to include index in XML document.
root_name (str, default ‘data’) – The name of root element in XML document.
row_name (str, default ‘row’) – The name of row element in XML document.
na_rep (str, optional) – Missing data representation.
attr_cols (list-like, optional) – List of columns to write as attributes in row element. Hierarchical columns will be flattened with underscore delimiting the different levels.
elem_cols (list-like, optional) – List of columns to write as children in row element. By default, all columns output as children of row element. Hierarchical columns will be flattened with underscore delimiting the different levels.
namespaces (dict, optional) – All namespaces to be defined in root element. Keys of dict should be prefix names and values of dict corresponding URIs. Default namespaces should be given empty string key. For example,
namespaces = {"": "https://example.com"}
prefix (str, optional) – Namespace prefix to be used for every element and/or attribute in document. This should be one of the keys in
namespaces
dict.encoding (str, default ‘utf-8’) – Encoding of the resulting document.
xml_declaration (bool, default True) – Whether to include the XML declaration at start of document.
pretty_print (bool, default True) – Whether output should be pretty printed with indentation and line breaks.
parser ({‘lxml’,’etree’}, default ‘lxml’) – Parser module to use for building of tree. Only ‘lxml’ and ‘etree’ are supported. With ‘lxml’, the ability to use XSLT stylesheet is supported.
stylesheet (str, path object or file-like object, optional) – A URL, file-like object, or a raw string containing an XSLT script used to transform the raw XML output. Script should use layout of elements and attributes from original output. This argument requires
lxml
to be installed. Only XSLT 1.0 scripts and not later versions is currently supported.compression (str or dict, default ‘infer’) – For on-the-fly compression of the output data. If ‘infer’ and ‘path_or_buffer’ path-like, then detect compression from the following extensions: ‘.gz’, ‘.bz2’, ‘.zip’, ‘.xz’, or ‘.zst’ (otherwise no compression). Set to
None
for no compression. Can also be a dict with key'method'
set to one of {'zip'
,'gzip'
,'bz2'
,'zstd'
} and other key-value pairs are forwarded tozipfile.ZipFile
,gzip.GzipFile
,bz2.BZ2File
, orzstandard.ZstdDecompressor
, respectively. As an example, the following could be passed for faster compression and to create a reproducible gzip archive:compression={'method': 'gzip', 'compresslevel': 1, 'mtime': 1}
.Changed in version 1.4.0: Zstandard support.
storage_options (dict, optional) – Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc. For HTTP(S) URLs the key-value pairs are forwarded to
urllib
as header options. For other URLs (e.g. starting with “s3://”, and “gcs://”) the key-value pairs are forwarded tofsspec
. Please seefsspec
andurllib
for more details.
- Returns
If
io
is None, returns the resulting XML format as a string. Otherwise returns None.- Return type
None or str
See also
to_json
Convert the pandas object to a JSON string.
to_html
Convert DataFrame to a html.
Examples
>>> df = pd.DataFrame({'shape': ['square', 'circle', 'triangle'], ... 'degrees': [360, 360, 180], ... 'sides': [4, np.nan, 3]})
>>> df.to_xml() <?xml version='1.0' encoding='utf-8'?> <data> <row> <index>0</index> <shape>square</shape> <degrees>360</degrees> <sides>4.0</sides> </row> <row> <index>1</index> <shape>circle</shape> <degrees>360</degrees> <sides/> </row> <row> <index>2</index> <shape>triangle</shape> <degrees>180</degrees> <sides>3.0</sides> </row> </data>
>>> df.to_xml(attr_cols=[ ... 'index', 'shape', 'degrees', 'sides' ... ]) <?xml version='1.0' encoding='utf-8'?> <data> <row index="0" shape="square" degrees="360" sides="4.0"/> <row index="1" shape="circle" degrees="360"/> <row index="2" shape="triangle" degrees="180" sides="3.0"/> </data>
>>> df.to_xml(namespaces={"doc": "https://example.com"}, ... prefix="doc") <?xml version='1.0' encoding='utf-8'?> <doc:data xmlns:doc="https://example.com"> <doc:row> <doc:index>0</doc:index> <doc:shape>square</doc:shape> <doc:degrees>360</doc:degrees> <doc:sides>4.0</doc:sides> </doc:row> <doc:row> <doc:index>1</doc:index> <doc:shape>circle</doc:shape> <doc:degrees>360</doc:degrees> <doc:sides/> </doc:row> <doc:row> <doc:index>2</doc:index> <doc:shape>triangle</doc:shape> <doc:degrees>180</doc:degrees> <doc:sides>3.0</doc:sides> </doc:row> </doc:data>
- transform(func, axis=0, *args, **kwargs)[source]¶
Call
func
on self producing a DataFrame with the same axis shape as self.- Parameters
func (function, str, list-like or dict-like) – Function to use for transforming the data. If a function, must either work when passed a DataFrame or when passed to DataFrame.apply. If func is both list-like and dict-like, dict-like behavior takes precedence.
Accepted combinations are:
function
string function name
list-like of functions and/or function names, e.g.
[np.exp, 'sqrt']
dict-like of axis labels -> functions, function names or list-like of such.
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – If 0 or ‘index’: apply function to each column. If 1 or ‘columns’: apply function to each row.
*args – Positional arguments to pass to func.
**kwargs – Keyword arguments to pass to func.
- Returns
A DataFrame that must have the same length as self.
- Return type
:raises ValueError : If the returned DataFrame has a different length than self.:
See also
DataFrame.agg
Only perform aggregating type operations.
DataFrame.apply
Invoke function on a DataFrame.
Notes
Functions that mutate the passed object can produce unexpected behavior or errors and are not supported. See Mutating with User Defined Function (UDF) methods for more details.
Examples
>>> df = pd.DataFrame({'A': range(3), 'B': range(1, 4)}) >>> df A B 0 0 1 1 1 2 2 2 3 >>> df.transform(lambda x: x + 1) A B 0 1 2 1 2 3 2 3 4
Even though the resulting DataFrame must have the same length as the input DataFrame, it is possible to provide several input functions:
>>> s = pd.Series(range(3)) >>> s 0 0 1 1 2 2 dtype: int64 >>> s.transform([np.sqrt, np.exp]) sqrt exp 0 0.000000 1.000000 1 1.000000 2.718282 2 1.414214 7.389056
You can call transform on a GroupBy object:
>>> df = pd.DataFrame({ ... "Date": [ ... "2015-05-08", "2015-05-07", "2015-05-06", "2015-05-05", ... "2015-05-08", "2015-05-07", "2015-05-06", "2015-05-05"], ... "Data": [5, 8, 6, 1, 50, 100, 60, 120], ... }) >>> df Date Data 0 2015-05-08 5 1 2015-05-07 8 2 2015-05-06 6 3 2015-05-05 1 4 2015-05-08 50 5 2015-05-07 100 6 2015-05-06 60 7 2015-05-05 120 >>> df.groupby('Date')['Data'].transform('sum') 0 55 1 108 2 66 3 121 4 55 5 108 6 66 7 121 Name: Data, dtype: int64
>>> df = pd.DataFrame({ ... "c": [1, 1, 1, 2, 2, 2, 2], ... "type": ["m", "n", "o", "m", "m", "n", "n"] ... }) >>> df c type 0 1 m 1 1 n 2 1 o 3 2 m 4 2 m 5 2 n 6 2 n >>> df['size'] = df.groupby('c')['type'].transform(len) >>> df c type size 0 1 m 3 1 1 n 3 2 1 o 3 3 2 m 4 4 2 m 4 5 2 n 4 6 2 n 4
- transpose(*args, copy=False)[source]¶
Transpose index and columns.
Reflect the DataFrame over its main diagonal by writing rows as columns and vice-versa. The property
T
is an accessor to the methodtranspose()
.- Parameters
*args (tuple, optional) – Accepted for compatibility with NumPy.
copy (bool, default False) – Whether to copy the data after transposing, even for DataFrames with a single dtype.
Note that a copy is always required for mixed dtype DataFrames, or for DataFrames with any extension types.
- Returns
The transposed DataFrame.
- Return type
See also
numpy.transpose
Permute the dimensions of a given array.
Notes
Transposing a DataFrame with mixed dtypes will result in a homogeneous DataFrame with the object dtype. In such a case, a copy of the data is always made.
Examples
Square DataFrame with homogeneous dtype
>>> d1 = {'col1': [1, 2], 'col2': [3, 4]} >>> df1 = pd.DataFrame(data=d1) >>> df1 col1 col2 0 1 3 1 2 4
>>> df1_transposed = df1.T # or df1.transpose() >>> df1_transposed 0 1 col1 1 2 col2 3 4
When the dtype is homogeneous in the original DataFrame, we get a transposed DataFrame with the same dtype:
>>> df1.dtypes col1 int64 col2 int64 dtype: object >>> df1_transposed.dtypes 0 int64 1 int64 dtype: object
Non-square DataFrame with mixed dtypes
>>> d2 = {'name': ['Alice', 'Bob'], ... 'score': [9.5, 8], ... 'employed': [False, True], ... 'kids': [0, 0]} >>> df2 = pd.DataFrame(data=d2) >>> df2 name score employed kids 0 Alice 9.5 False 0 1 Bob 8.0 True 0
>>> df2_transposed = df2.T # or df2.transpose() >>> df2_transposed 0 1 name Alice Bob score 9.5 8.0 employed False True kids 0 0
When the DataFrame has mixed dtypes, we get a transposed DataFrame with the object dtype:
>>> df2.dtypes name object score float64 employed bool kids int64 dtype: object >>> df2_transposed.dtypes 0 object 1 object dtype: object
- truediv(other, axis='columns', level=None, fill_value=None)¶
Get Floating division of dataframe and other, element-wise (binary operator truediv).
Equivalent to
dataframe / other
, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, rtruediv.Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.
- Parameters
other (scalar, sequence, Series, or DataFrame) – Any single or multiple element data structure, or list-like object.
axis ({0 or ‘index’, 1 or ‘columns’}) – Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.
level (int or label) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (float or None, default None) – Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.
- Returns
Result of the arithmetic operation.
- Return type
See also
DataFrame.add
Add DataFrames.
DataFrame.sub
Subtract DataFrames.
DataFrame.mul
Multiply DataFrames.
DataFrame.div
Divide DataFrames (float division).
DataFrame.truediv
Divide DataFrames (float division).
DataFrame.floordiv
Divide DataFrames (integer division).
DataFrame.mod
Calculate modulo (remainder after division).
DataFrame.pow
Calculate exponential power.
Notes
Mismatched indices will be unioned together.
Examples
>>> df = pd.DataFrame({'angles': [0, 3, 4], ... 'degrees': [360, 180, 360]}, ... index=['circle', 'triangle', 'rectangle']) >>> df angles degrees circle 0 360 triangle 3 180 rectangle 4 360
Add a scalar with operator version which return the same results.
>>> df + 1 angles degrees circle 1 361 triangle 4 181 rectangle 5 361
>>> df.add(1) angles degrees circle 1 361 triangle 4 181 rectangle 5 361
Divide by constant with reverse version.
>>> df.div(10) angles degrees circle 0.0 36.0 triangle 0.3 18.0 rectangle 0.4 36.0
>>> df.rdiv(10) angles degrees circle inf 0.027778 triangle 3.333333 0.055556 rectangle 2.500000 0.027778
Subtract a list and Series by axis with operator version.
>>> df - [1, 2] angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub([1, 2], axis='columns') angles degrees circle -1 358 triangle 2 178 rectangle 3 358
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']), ... axis='index') angles degrees circle -1 359 triangle 2 179 rectangle 3 359
Multiply a DataFrame of different shape with operator version.
>>> other = pd.DataFrame({'angles': [0, 3, 4]}, ... index=['circle', 'triangle', 'rectangle']) >>> other angles circle 0 triangle 3 rectangle 4
>>> df * other angles degrees circle 0 NaN triangle 9 NaN rectangle 16 NaN
>>> df.mul(other, fill_value=0) angles degrees circle 0 0.0 triangle 9 0.0 rectangle 16 0.0
Divide by a MultiIndex by level.
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6], ... 'degrees': [360, 180, 360, 360, 540, 720]}, ... index=[['A', 'A', 'A', 'B', 'B', 'B'], ... ['circle', 'triangle', 'rectangle', ... 'square', 'pentagon', 'hexagon']]) >>> df_multindex angles degrees A circle 0 360 triangle 3 180 rectangle 4 360 B square 4 360 pentagon 5 540 hexagon 6 720
>>> df.div(df_multindex, level=1, fill_value=0) angles degrees A circle NaN 1.0 triangle 1.0 1.0 rectangle 1.0 1.0 B square 0.0 0.0 pentagon 0.0 0.0 hexagon 0.0 0.0
- unstack(level=- 1, fill_value=None)[source]¶
Pivot a level of the (necessarily hierarchical) index labels.
Returns a DataFrame having a new level of column labels whose inner-most level consists of the pivoted index labels.
If the index is not a MultiIndex, the output will be a Series (the analogue of stack when the columns are not a MultiIndex).
- Parameters
level (int, str, or list of these, default -1 (last level)) – Level(s) of index to unstack, can pass level name.
fill_value (int, str or dict) – Replace NaN with this value if the unstack produces missing values.
- Return type
See also
DataFrame.pivot
Pivot a table based on column values.
DataFrame.stack
Pivot a level of the column labels (inverse operation from unstack).
Notes
Reference the user guide for more examples.
Examples
>>> index = pd.MultiIndex.from_tuples([('one', 'a'), ('one', 'b'), ... ('two', 'a'), ('two', 'b')]) >>> s = pd.Series(np.arange(1.0, 5.0), index=index) >>> s one a 1.0 b 2.0 two a 3.0 b 4.0 dtype: float64
>>> s.unstack(level=-1) a b one 1.0 2.0 two 3.0 4.0
>>> s.unstack(level=0) one two a 1.0 3.0 b 2.0 4.0
>>> df = s.unstack(level=0) >>> df.unstack() one a 1.0 b 2.0 two a 3.0 b 4.0 dtype: float64
- update(other, join='left', overwrite=True, filter_func=None, errors='ignore')[source]¶
Modify in place using non-NA values from another DataFrame.
Aligns on indices. There is no return value.
- Parameters
other (DataFrame, or object coercible into a DataFrame) – Should have at least one matching index/column label with the original DataFrame. If a Series is passed, its name attribute must be set, and that will be used as the column name to align with the original DataFrame.
join ({‘left’}, default ‘left’) – Only left join is implemented, keeping the index and columns of the original object.
overwrite (bool, default True) – How to handle non-NA values for overlapping keys:
True: overwrite original DataFrame’s values with values from other.
False: only update values that are NA in the original DataFrame.
filter_func (callable(1d-array) -> bool 1d-array, optional) – Can choose to replace values other than NA. Return True for values that should be updated.
errors ({‘raise’, ‘ignore’}, default ‘ignore’) – If ‘raise’, will raise a ValueError if the DataFrame and other both contain non-NA data in the same place.
- Returns
None
- Return type
method directly changes calling object
- Raises
When errors=’raise’ and there’s overlapping non-NA data. * When errors is not either ‘ignore’ or ‘raise’
If join != ‘left’
See also
dict.update
Similar method for dictionaries.
DataFrame.merge
For column(s)-on-column(s) operations.
Examples
>>> df = pd.DataFrame({'A': [1, 2, 3], ... 'B': [400, 500, 600]}) >>> new_df = pd.DataFrame({'B': [4, 5, 6], ... 'C': [7, 8, 9]}) >>> df.update(new_df) >>> df A B 0 1 4 1 2 5 2 3 6
The DataFrame’s length does not increase as a result of the update, only values at matching index/column labels are updated.
>>> df = pd.DataFrame({'A': ['a', 'b', 'c'], ... 'B': ['x', 'y', 'z']}) >>> new_df = pd.DataFrame({'B': ['d', 'e', 'f', 'g', 'h', 'i']}) >>> df.update(new_df) >>> df A B 0 a d 1 b e 2 c f
For Series, its name attribute must be set.
>>> df = pd.DataFrame({'A': ['a', 'b', 'c'], ... 'B': ['x', 'y', 'z']}) >>> new_column = pd.Series(['d', 'e'], name='B', index=[0, 2]) >>> df.update(new_column) >>> df A B 0 a d 1 b y 2 c e >>> df = pd.DataFrame({'A': ['a', 'b', 'c'], ... 'B': ['x', 'y', 'z']}) >>> new_df = pd.DataFrame({'B': ['d', 'e']}, index=[1, 2]) >>> df.update(new_df) >>> df A B 0 a x 1 b d 2 c e
If other contains NaNs the corresponding values are not updated in the original dataframe.
>>> df = pd.DataFrame({'A': [1, 2, 3], ... 'B': [400, 500, 600]}) >>> new_df = pd.DataFrame({'B': [4, np.nan, 6]}) >>> df.update(new_df) >>> df A B 0 1 4.0 1 2 500.0 2 3 6.0
- value_counts(subset=None, normalize=False, sort=True, ascending=False, dropna=True)[source]¶
Return a Series containing counts of unique rows in the DataFrame.
New in version 1.1.0.
- Parameters
subset (list-like, optional) – Columns to use when counting unique combinations.
normalize (bool, default False) – Return proportions rather than frequencies.
sort (bool, default True) – Sort by frequencies.
ascending (bool, default False) – Sort in ascending order.
dropna (bool, default True) – Don’t include counts of rows that contain NA values.
New in version 1.3.0.
- Return type
See also
Series.value_counts
Equivalent method on Series.
Notes
The returned Series will have a MultiIndex with one level per input column. By default, rows that contain any NA values are omitted from the result. By default, the resulting Series will be in descending order so that the first element is the most frequently-occurring row.
Examples
>>> df = pd.DataFrame({'num_legs': [2, 4, 4, 6], ... 'num_wings': [2, 0, 0, 0]}, ... index=['falcon', 'dog', 'cat', 'ant']) >>> df num_legs num_wings falcon 2 2 dog 4 0 cat 4 0 ant 6 0
>>> df.value_counts() num_legs num_wings 4 0 2 2 2 1 6 0 1 dtype: int64
>>> df.value_counts(sort=False) num_legs num_wings 2 2 1 4 0 2 6 0 1 dtype: int64
>>> df.value_counts(ascending=True) num_legs num_wings 2 2 1 6 0 1 4 0 2 dtype: int64
>>> df.value_counts(normalize=True) num_legs num_wings 4 0 0.50 2 2 0.25 6 0 0.25 dtype: float64
With dropna set to False we can also count rows with NA values.
>>> df = pd.DataFrame({'first_name': ['John', 'Anne', 'John', 'Beth'], ... 'middle_name': ['Smith', pd.NA, pd.NA, 'Louise']}) >>> df first_name middle_name 0 John Smith 1 Anne <NA> 2 John <NA> 3 Beth Louise
>>> df.value_counts() first_name middle_name Beth Louise 1 John Smith 1 dtype: int64
>>> df.value_counts(dropna=False) first_name middle_name Anne NaN 1 Beth Louise 1 John Smith 1 NaN 1 dtype: int64
- property values: numpy.ndarray¶
Return a Numpy representation of the DataFrame.
Warning
We recommend using
DataFrame.to_numpy()
instead.Only the values in the DataFrame will be returned, the axes labels will be removed.
- Returns
The values of the DataFrame.
- Return type
See also
DataFrame.to_numpy
Recommended alternative to this method.
DataFrame.index
Retrieve the index labels.
DataFrame.columns
Retrieving the column names.
Notes
The dtype will be a lower-common-denominator dtype (implicit upcasting); that is to say if the dtypes (even of numeric types) are mixed, the one that accommodates all will be chosen. Use this with care if you are not dealing with the blocks.
e.g. If the dtypes are float16 and float32, dtype will be upcast to float32. If dtypes are int32 and uint8, dtype will be upcast to int32. By
numpy.find_common_type()
convention, mixing int64 and uint64 will result in a float64 dtype.Examples
A DataFrame where all columns are the same type (e.g., int64) results in an array of the same type.
>>> df = pd.DataFrame({'age': [ 3, 29], ... 'height': [94, 170], ... 'weight': [31, 115]}) >>> df age height weight 0 3 94 31 1 29 170 115 >>> df.dtypes age int64 height int64 weight int64 dtype: object >>> df.values array([[ 3, 94, 31], [ 29, 170, 115]])
A DataFrame with mixed type columns(e.g., str/object, int64, float32) results in an ndarray of the broadest type that accommodates these mixed types (e.g., object).
>>> df2 = pd.DataFrame([('parrot', 24.0, 'second'), ... ('lion', 80.5, 1), ... ('monkey', np.nan, None)], ... columns=('name', 'max_speed', 'rank')) >>> df2.dtypes name object max_speed float64 rank object dtype: object >>> df2.values array([['parrot', 24.0, 'second'], ['lion', 80.5, 1], ['monkey', nan, None]], dtype=object)
- var(axis=None, skipna=True, level=None, ddof=1, numeric_only=None, **kwargs)¶
Return unbiased variance over requested axis.
Normalized by N-1 by default. This can be changed using the ddof argument.
- Parameters
axis ({index (0), columns (1)})
skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a Series.
ddof (int, default 1) – Delta Degrees of Freedom. The divisor used in calculations is N - ddof, where N represents the number of elements.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
- Return type
Examples
>>> df = pd.DataFrame({'person_id': [0, 1, 2, 3], ... 'age': [21, 25, 62, 43], ... 'height': [1.61, 1.87, 1.49, 2.01]} ... ).set_index('person_id') >>> df age height person_id 0 21 1.61 1 25 1.87 2 62 1.49 3 43 2.01
>>> df.var() age 352.916667 height 0.056367
Alternatively,
ddof=0
can be set to normalize by N instead of N-1:>>> df.var(ddof=0) age 264.687500 height 0.042275
- where(cond, other=NoDefault.no_default, inplace=False, axis=None, level=None, errors='raise', try_cast=NoDefault.no_default)[source]¶
Replace values where the condition is False.
- Parameters
cond (bool Series/DataFrame, array-like, or callable) – Where cond is True, keep the original value. Where False, replace with corresponding value from other. If cond is callable, it is computed on the Series/DataFrame and should return boolean Series/DataFrame or array. The callable must not change input Series/DataFrame (though pandas doesn’t check it).
other (scalar, Series/DataFrame, or callable) – Entries where cond is False are replaced with corresponding value from other. If other is callable, it is computed on the Series/DataFrame and should return scalar or Series/DataFrame. The callable must not change input Series/DataFrame (though pandas doesn’t check it).
inplace (bool, default False) – Whether to perform the operation in place on the data.
axis (int, default None) – Alignment axis if needed.
level (int, default None) – Alignment level if needed.
errors (str, {‘raise’, ‘ignore’}, default ‘raise’) – Note that currently this parameter won’t affect the results and will always coerce to a suitable dtype.
‘raise’ : allow exceptions to be raised.
‘ignore’ : suppress exceptions. On error return original object.
try_cast (bool, default None) – Try to cast the result back to the input type (if possible).
Deprecated since version 1.3.0: Manually cast back if necessary.
- Return type
Same type as caller or None if
inplace=True
.
See also
DataFrame.mask()
Return an object of same shape as self.
Notes
The where method is an application of the if-then idiom. For each element in the calling DataFrame, if
cond
isTrue
the element is used; otherwise the corresponding element from the DataFrameother
is used.The signature for
DataFrame.where()
differs fromnumpy.where()
. Roughlydf1.where(m, df2)
is equivalent tonp.where(m, df1, df2)
.For further details and examples see the
where
documentation in indexing.Examples
>>> s = pd.Series(range(5)) >>> s.where(s > 0) 0 NaN 1 1.0 2 2.0 3 3.0 4 4.0 dtype: float64 >>> s.mask(s > 0) 0 0.0 1 NaN 2 NaN 3 NaN 4 NaN dtype: float64
>>> s.where(s > 1, 10) 0 10 1 10 2 2 3 3 4 4 dtype: int64 >>> s.mask(s > 1, 10) 0 0 1 1 2 10 3 10 4 10 dtype: int64
>>> df = pd.DataFrame(np.arange(10).reshape(-1, 2), columns=['A', 'B']) >>> df A B 0 0 1 1 2 3 2 4 5 3 6 7 4 8 9 >>> m = df % 3 == 0 >>> df.where(m, -df) A B 0 0 -1 1 -2 3 2 -4 -5 3 6 -7 4 -8 9 >>> df.where(m, -df) == np.where(m, df, -df) A B 0 True True 1 True True 2 True True 3 True True 4 True True >>> df.where(m, -df) == df.mask(~m, -df) A B 0 True True 1 True True 2 True True 3 True True 4 True True
- class imc.types.Figure(figsize=None, dpi=None, facecolor=None, edgecolor=None, linewidth=0.0, frameon=None, subplotpars=None, tight_layout=None, constrained_layout=None, *, layout=None, **kwargs)[source]¶
The top level container for all the plot elements.
The Figure instance supports callbacks through a callbacks attribute which is a .CallbackRegistry instance. The events you can connect to are ‘dpi_changed’, and the callback will be called with
func(fig)
where fig is the Figure instance.- Variables
~Figure.patch – The .Rectangle instance representing the figure background patch.
~Figure.suppressComposite – For multiple images, the figure will make composite images depending on the renderer option_image_nocomposite function. If suppressComposite is a boolean, this will override the renderer.
- property axes¶
List of Axes in the Figure. You can access and modify the Axes in the Figure through this list.
Do not modify the list itself. Instead, use ~Figure.add_axes, ~.Figure.add_subplot or ~.Figure.delaxes to add or remove an Axes.
Note: The .Figure.axes property and ~.Figure.get_axes method are equivalent.
- clf(keep_observers=False)[source]¶
Clear the figure.
Set keep_observers to True if, for example, a gui widget is tracking the Axes in the figure.
- property dpi¶
The resolution in dots per inch.
- draw(renderer)[source]¶
Draw the Artist (and its children) using the given renderer.
This has no effect if the artist is not visible (.Artist.get_visible returns False).
- Parameters
renderer (.RendererBase subclass.)
Notes
This method is overridden in the Artist subclasses.
- draw_artist(a)[source]¶
Draw .Artist a only.
This method can only be used after an initial draw of the figure, because that creates and caches the renderer needed here.
- draw_without_rendering()[source]¶
Draw the figure with no output. Useful to get the final size of artists that require a draw before their size is known (e.g. text).
- execute_constrained_layout(renderer=None)[source]¶
Use
layoutgrid
to determine pos positions within Axes.See also .set_constrained_layout_pads.
- Returns
layoutgrid
- Return type
private debugging object
- figimage(X, xo=0, yo=0, alpha=None, norm=None, cmap=None, vmin=None, vmax=None, origin=None, resize=False, **kwargs)[source]¶
Add a non-resampled image to the figure.
The image is attached to the lower or upper left corner depending on origin.
- Parameters
X – The image data. This is an array of one of the following shapes:
MxN: luminance (grayscale) values
MxNx3: RGB values
MxNx4: RGBA values
xo, yo (int) – The x/y image offset in pixels.
alpha (None or float) – The alpha blending value.
norm (matplotlib.colors.Normalize) – A .Normalize instance to map the luminance to the interval [0, 1].
cmap (str or matplotlib.colors.Colormap, default: :rc:`image.cmap`) – The colormap to use.
vmin, vmax (float) – If norm is not given, these values set the data limits for the colormap.
origin ({‘upper’, ‘lower’}, default: :rc:`image.origin`) – Indicates where the [0, 0] index of the array is in the upper left or lower left corner of the axes.
resize (bool) – If True, resize the figure to match the given image size.
- Return type
matplotlib.image.FigureImage
- Other Parameters
**kwargs – Additional kwargs are .Artist kwargs passed on to .FigureImage.
Notes
figimage complements the Axes image (~matplotlib.axes.Axes.imshow) which will be resampled to fit the current Axes. If you want a resampled image to fill the entire figure, you can define an ~matplotlib.axes.Axes with extent [0, 0, 1, 1].
Examples
f = plt.figure() nx = int(f.get_figwidth() * f.dpi) ny = int(f.get_figheight() * f.dpi) data = np.random.random((ny, nx)) f.figimage(data) plt.show()
- get_axes()¶
List of Axes in the Figure. You can access and modify the Axes in the Figure through this list.
Do not modify the list itself. Instead, use ~Figure.add_axes, ~.Figure.add_subplot or ~.Figure.delaxes to add or remove an Axes.
Note: The .Figure.axes property and ~.Figure.get_axes method are equivalent.
- get_constrained_layout()[source]¶
Return whether constrained layout is being used.
See /tutorials/intermediate/constrainedlayout_guide.
- get_constrained_layout_pads(relative=False)[source]¶
Get padding for
constrained_layout
.Returns a list of
w_pad, h_pad
in inches andwspace
andhspace
as fractions of the subplot.See /tutorials/intermediate/constrainedlayout_guide.
- Parameters
relative (bool) – If True, then convert from inches to figure relative.
- get_size_inches()[source]¶
Return the current size of the figure in inches.
- Returns
The size (width, height) of the figure in inches.
- Return type
ndarray
See also
matplotlib.figure.Figure.set_size_inches
,matplotlib.figure.Figure.get_figwidth
,matplotlib.figure.Figure.get_figheight
Notes
The size in pixels can be obtained by multiplying with Figure.dpi.
- ginput(n=1, timeout=30, show_clicks=True, mouse_add=MouseButton.LEFT, mouse_pop=MouseButton.RIGHT, mouse_stop=MouseButton.MIDDLE)[source]¶
Blocking call to interact with a figure.
Wait until the user clicks n times on the figure, and return the coordinates of each click in a list.
There are three possible interactions:
Add a point.
Remove the most recently added point.
Stop the interaction and return the points added so far.
The actions are assigned to mouse buttons via the arguments mouse_add, mouse_pop and mouse_stop.
- Parameters
n (int, default: 1) – Number of mouse clicks to accumulate. If negative, accumulate clicks until the input is terminated manually.
timeout (float, default: 30 seconds) – Number of seconds to wait before timing out. If zero or negative will never timeout.
show_clicks (bool, default: True) – If True, show a red cross at the location of each click.
mouse_add (.MouseButton or None, default: .MouseButton.LEFT) – Mouse button used to add points.
mouse_pop (.MouseButton or None, default: .MouseButton.RIGHT) – Mouse button used to remove the most recently added point.
mouse_stop (.MouseButton or None, default: .MouseButton.MIDDLE) – Mouse button used to stop input.
- Returns
A list of the clicked (x, y) coordinates.
- Return type
list of tuples
Notes
The keyboard can also be used to select points in case your mouse does not have one or more of the buttons. The delete and backspace keys act like right clicking (i.e., remove last point), the enter key terminates input and any other key (not already used by the window manager) selects a point.
- savefig(fname, *, transparent=None, **kwargs)[source]¶
Save the current figure.
Call signature:
savefig(fname, *, dpi='figure', format=None, metadata=None, bbox_inches=None, pad_inches=0.1, facecolor='auto', edgecolor='auto', backend=None, **kwargs )
The available output formats depend on the backend being used.
- Parameters
fname (str or path-like or binary file-like) – A path, or a Python file-like object, or possibly some backend-dependent object such as matplotlib.backends.backend_pdf.PdfPages.
If format is set, it determines the output format, and the file is saved as fname. Note that fname is used verbatim, and there is no attempt to make the extension, if any, of fname match format, and no extension is appended.
If format is not set, then the format is inferred from the extension of fname, if there is one. If format is not set and fname has no extension, then the file is saved with :rc:`savefig.format` and the appropriate extension is appended to fname.
- Other Parameters
dpi (float or ‘figure’, default: :rc:`savefig.dpi`) – The resolution in dots per inch. If ‘figure’, use the figure’s dpi value.
format (str) – The file format, e.g. ‘png’, ‘pdf’, ‘svg’, … The behavior when this is unset is documented under fname.
metadata (dict, optional) – Key/value pairs to store in the image metadata. The supported keys and defaults depend on the image format and backend:
‘png’ with Agg backend: See the parameter
metadata
of ~.FigureCanvasAgg.print_png.‘pdf’ with pdf backend: See the parameter
metadata
of ~.backend_pdf.PdfPages.‘svg’ with svg backend: See the parameter
metadata
of ~.FigureCanvasSVG.print_svg.‘eps’ and ‘ps’ with PS backend: Only ‘Creator’ is supported.
bbox_inches (str or .Bbox, default: :rc:`savefig.bbox`) – Bounding box in inches: only the given portion of the figure is saved. If ‘tight’, try to figure out the tight bbox of the figure.
pad_inches (float, default: :rc:`savefig.pad_inches`) – Amount of padding around the figure when bbox_inches is ‘tight’.
facecolor (color or ‘auto’, default: :rc:`savefig.facecolor`) – The facecolor of the figure. If ‘auto’, use the current figure facecolor.
edgecolor (color or ‘auto’, default: :rc:`savefig.edgecolor`) – The edgecolor of the figure. If ‘auto’, use the current figure edgecolor.
backend (str, optional) – Use a non-default backend to render the file, e.g. to render a png file with the “cairo” backend rather than the default “agg”, or a pdf file with the “pgf” backend rather than the default “pdf”. Note that the default backend is normally sufficient. See the-builtin-backends for a list of valid backends for each file format. Custom backends can be referenced as “module://…”.
orientation ({‘landscape’, ‘portrait’}) – Currently only supported by the postscript backend.
papertype (str) – One of ‘letter’, ‘legal’, ‘executive’, ‘ledger’, ‘a0’ through ‘a10’, ‘b0’ through ‘b10’. Only supported for postscript output.
transparent (bool) – If True, the Axes patches will all be transparent; the Figure patch will also be transparent unless facecolor and/or edgecolor are specified via kwargs.
If False has no effect and the color of the Axes and Figure patches are unchanged (unless the Figure patch is specified via the facecolor and/or edgecolor keyword arguments in which case those colors are used).
The transparency of these patches will be restored to their original values upon exit of this function.
This is useful, for example, for displaying a plot on top of a colored background on a web page.
bbox_extra_artists (list of ~matplotlib.artist.Artist, optional) – A list of extra artists that will be considered when the tight bbox is calculated.
pil_kwargs (dict, optional) – Additional keyword arguments that are passed to PIL.Image.Image.save when saving the figure.
- set(*, agg_filter=<UNSET>, alpha=<UNSET>, animated=<UNSET>, canvas=<UNSET>, clip_box=<UNSET>, clip_on=<UNSET>, clip_path=<UNSET>, constrained_layout=<UNSET>, constrained_layout_pads=<UNSET>, dpi=<UNSET>, edgecolor=<UNSET>, facecolor=<UNSET>, figheight=<UNSET>, figwidth=<UNSET>, frameon=<UNSET>, gid=<UNSET>, in_layout=<UNSET>, label=<UNSET>, linewidth=<UNSET>, path_effects=<UNSET>, picker=<UNSET>, rasterized=<UNSET>, size_inches=<UNSET>, sketch_params=<UNSET>, snap=<UNSET>, tight_layout=<UNSET>, transform=<UNSET>, url=<UNSET>, visible=<UNSET>, zorder=<UNSET>)¶
Set multiple properties at once.
Supported properties are
- Properties:
agg_filter: a filter function, which takes a (m, n, 3) float array and a dpi value, and returns a (m, n, 3) array alpha: scalar or None animated: bool canvas: FigureCanvas clip_box: .Bbox clip_on: bool clip_path: Patch or (Path, Transform) or None constrained_layout: bool or dict or None constrained_layout_pads: float, default: :rc:`figure.constrained_layout.w_pad` dpi: float edgecolor: color facecolor: color figheight: float figure: .Figure figwidth: float frameon: bool gid: str in_layout: bool label: object linewidth: number path_effects: .AbstractPathEffect picker: None or bool or float or callable rasterized: bool size_inches: (float, float) or float sketch_params: (scale: float, length: float, randomness: float) snap: bool or None tight_layout: bool or dict with keys “pad”, “w_pad”, “h_pad”, “rect” or None transform: .Transform url: str visible: bool zorder: float
- set_canvas(canvas)[source]¶
Set the canvas that contains the figure
- Parameters
canvas (FigureCanvas)
- set_constrained_layout(constrained)[source]¶
Set whether
constrained_layout
is used upon drawing. If None, :rc:`figure.constrained_layout.use` value will be used.When providing a dict containing the keys w_pad, h_pad the default
constrained_layout
paddings will be overridden. These pads are in inches and default to 3.0/72.0.w_pad
is the width padding andh_pad
is the height padding.See /tutorials/intermediate/constrainedlayout_guide.
- Parameters
constrained (bool or dict or None)
- set_constrained_layout_pads(*, w_pad=None, h_pad=None, wspace=None, hspace=None)[source]¶
Set padding for
constrained_layout
.Tip: The parameters can be passed from a dictionary by using
fig.set_constrained_layout(**pad_dict)
.See /tutorials/intermediate/constrainedlayout_guide.
- Parameters
w_pad (float, default: :rc:`figure.constrained_layout.w_pad`) – Width padding in inches. This is the pad around Axes and is meant to make sure there is enough room for fonts to look good. Defaults to 3 pts = 0.04167 inches
h_pad (float, default: :rc:`figure.constrained_layout.h_pad`) – Height padding in inches. Defaults to 3 pts.
wspace (float, default: :rc:`figure.constrained_layout.wspace`) – Width padding between subplots, expressed as a fraction of the subplot width. The total padding ends up being w_pad + wspace.
hspace (float, default: :rc:`figure.constrained_layout.hspace`) – Height padding between subplots, expressed as a fraction of the subplot width. The total padding ends up being h_pad + hspace.
- set_figheight(val, forward=True)[source]¶
Set the height of the figure in inches.
- Parameters
val (float)
forward (bool) – See set_size_inches.
See also
matplotlib.figure.Figure.set_figwidth
,matplotlib.figure.Figure.set_size_inches
- set_figwidth(val, forward=True)[source]¶
Set the width of the figure in inches.
- Parameters
val (float)
forward (bool) – See set_size_inches.
See also
matplotlib.figure.Figure.set_figheight
,matplotlib.figure.Figure.set_size_inches
- set_size_inches(w, h=None, forward=True)[source]¶
Set the figure size in inches.
Call signatures:
fig.set_size_inches(w, h) # OR fig.set_size_inches((w, h))
- Parameters
w ((float, float) or float) – Width and height in inches (if height not specified as a separate argument) or width.
h (float) – Height in inches.
forward (bool, default: True) – If
True
, the canvas size is automatically updated, e.g., you can resize the figure window from the shell.
See also
matplotlib.figure.Figure.get_size_inches
,matplotlib.figure.Figure.set_figwidth
,matplotlib.figure.Figure.set_figheight
Notes
To transform from pixels to inches divide by Figure.dpi.
- set_tight_layout(tight)[source]¶
Set whether and how .tight_layout is called when drawing.
- Parameters
tight (bool or dict with keys “pad”, “w_pad”, “h_pad”, “rect” or None) – If a bool, sets whether to call .tight_layout upon drawing. If
None
, use :rc:`figure.autolayout` instead. If a dict, pass it as kwargs to .tight_layout, overriding the default paddings.
- show(warn=True)[source]¶
If using a GUI backend with pyplot, display the figure window.
If the figure was not created using ~.pyplot.figure, it will lack a ~.backend_bases.FigureManagerBase, and this method will raise an AttributeError.
Warning
This does not manage an GUI event loop. Consequently, the figure may only be shown briefly or not shown at all if you or your environment are not managing an event loop.
Proper use cases for .Figure.show include running this from a GUI application or an IPython shell.
If you’re running a pure python shell or executing a non-GUI python script, you should use matplotlib.pyplot.show instead, which takes care of managing the event loop for you.
- Parameters
warn (bool, default: True) – If
True
and we are not running headless (i.e. on Linux with an unset DISPLAY), issue warning when called on a non-GUI backend.
- tight_layout(*, pad=1.08, h_pad=None, w_pad=None, rect=None)[source]¶
Adjust the padding between and around subplots.
To exclude an artist on the Axes from the bounding box calculation that determines the subplot parameters (i.e. legend, or annotation), set
a.set_in_layout(False)
for that artist.- Parameters
pad (float, default: 1.08) – Padding between the figure edge and the edges of subplots, as a fraction of the font size.
h_pad, w_pad (float, default: *pad*) – Padding (height/width) between edges of adjacent subplots, as a fraction of the font size.
rect (tuple (left, bottom, right, top), default: (0, 0, 1, 1)) – A rectangle in normalized figure coordinates into which the whole subplots area (including labels) will fit.
See also
Figure.set_tight_layout
,pyplot.tight_layout
- imc.types.MultiIndexSeries¶
alias of
pandas.core.series.Series
- class imc.types.Patch(edgecolor=None, facecolor=None, color=None, linewidth=None, linestyle=None, antialiased=None, hatch=None, fill=True, capstyle=None, joinstyle=None, **kwargs)[source]¶
A patch is a 2D artist with a face color and an edge color.
If any of edgecolor, facecolor, linewidth, or antialiased are None, they default to their rc params setting.
- contains(mouseevent, radius=None)[source]¶
Test whether the mouse event occurred in the patch.
- Return type
(bool, empty dict)
- contains_point(point, radius=None)[source]¶
Return whether the given point is inside the patch.
- Parameters
point ((float, float)) – The point (x, y) to check, in target coordinates of
self.get_transform()
. These are display coordinates for patches that are added to a figure or axes.radius (float, optional) – Add an additional margin on the patch in target coordinates of
self.get_transform()
. See .Path.contains_point for further details.
- Return type
Notes
The proper use of this method depends on the transform of the patch. Isolated patches do not have a transform. In this case, the patch creation coordinates and the point coordinates match. The following example checks that the center of a circle is within the circle
>>> center = 0, 0 >>> c = Circle(center, radius=1) >>> c.contains_point(center) True
The convention of checking against the transformed patch stems from the fact that this method is predominantly used to check if display coordinates (e.g. from mouse events) are within the patch. If you want to do the above check with data coordinates, you have to properly transform them first:
>>> center = 0, 0 >>> c = Circle(center, radius=1) >>> plt.gca().add_patch(c) >>> transformed_center = c.get_transform().transform(center) >>> c.contains_point(transformed_center) True
- contains_points(points, radius=None)[source]¶
Return whether the given points are inside the patch.
- Parameters
points ((N, 2) array) – The points to check, in target coordinates of
self.get_transform()
. These are display coordinates for patches that are added to a figure or axes. Columns contain x and y values.radius (float, optional) – Add an additional margin on the patch in target coordinates of
self.get_transform()
. See .Path.contains_point for further details.
- Return type
length-N bool array
Notes
The proper use of this method depends on the transform of the patch. See the notes on .Patch.contains_point.
- draw(renderer)[source]¶
Draw the Artist (and its children) using the given renderer.
This has no effect if the artist is not visible (.Artist.get_visible returns False).
- Parameters
renderer (.RendererBase subclass.)
Notes
This method is overridden in the Artist subclasses.
- property fill¶
Return whether the patch is filled.
- get_aa()¶
Alias for get_antialiased.
- get_data_transform()[source]¶
Return the ~.transforms.Transform mapping data coordinates to physical coordinates.
- get_ec()¶
Alias for get_edgecolor.
- get_fc()¶
Alias for get_facecolor.
- get_ls()¶
Alias for get_linestyle.
- get_lw()¶
Alias for get_linewidth.
- get_patch_transform()[source]¶
Return the ~.transforms.Transform instance mapping patch coordinates to data coordinates.
For example, one may define a patch of a circle which represents a radius of 5 by providing coordinates for a unit circle, and a transform which scales the coordinates (the patch coordinate) by 5.
- get_verts()[source]¶
Return a copy of the vertices used in this patch.
If the patch contains Bezier curves, the curves will be interpolated by line segments. To access the curves as curves, use get_path.
- get_window_extent(renderer=None)[source]¶
Get the artist’s bounding box in display space.
The bounding box’ width and height are nonnegative.
Subclasses should override for inclusion in the bounding box “tight” calculation. Default is to return an empty bounding box at 0, 0.
Be careful when using this function, the results will not update if the artist window extent of the artist changes. The extent can change due to any changes in the transform stack, such as changing the axes limits, the figure size, or the canvas used (as is done when saving a figure). This can lead to unexpected behavior where interactive figures will look fine on the screen, but will save incorrectly.
- set(*, agg_filter=<UNSET>, alpha=<UNSET>, animated=<UNSET>, antialiased=<UNSET>, capstyle=<UNSET>, clip_box=<UNSET>, clip_on=<UNSET>, clip_path=<UNSET>, color=<UNSET>, edgecolor=<UNSET>, facecolor=<UNSET>, fill=<UNSET>, gid=<UNSET>, hatch=<UNSET>, in_layout=<UNSET>, joinstyle=<UNSET>, label=<UNSET>, linestyle=<UNSET>, linewidth=<UNSET>, path_effects=<UNSET>, picker=<UNSET>, rasterized=<UNSET>, sketch_params=<UNSET>, snap=<UNSET>, transform=<UNSET>, url=<UNSET>, visible=<UNSET>, zorder=<UNSET>)¶
Set multiple properties at once.
Supported properties are
- Properties:
agg_filter: a filter function, which takes a (m, n, 3) float array and a dpi value, and returns a (m, n, 3) array alpha: unknown animated: bool antialiased: bool or None capstyle: .CapStyle or {‘butt’, ‘projecting’, ‘round’} clip_box: .Bbox clip_on: bool clip_path: Patch or (Path, Transform) or None color: color edgecolor: color or None facecolor: color or None figure: .Figure fill: bool gid: str hatch: {‘/’, ‘\’, ‘|’, ‘-’, ‘+’, ‘x’, ‘o’, ‘O’, ‘.’, ‘*’} in_layout: bool joinstyle: .JoinStyle or {‘miter’, ‘round’, ‘bevel’} label: object linestyle: {‘-’, ‘–’, ‘-.’, ‘:’, ‘’, (offset, on-off-seq), …} linewidth: float or None path_effects: .AbstractPathEffect picker: None or bool or float or callable rasterized: bool sketch_params: (scale: float, length: float, randomness: float) snap: bool or None transform: .Transform url: str visible: bool zorder: float
- set_aa(aa)¶
Alias for set_antialiased.
- set_alpha(alpha)[source]¶
Set the alpha value used for blending - not supported on all backends.
- Parameters
alpha (scalar or None) – alpha must be within the 0-1 range, inclusive.
- set_capstyle(s)[source]¶
Set the .CapStyle.
- Parameters
s (.CapStyle or {‘butt’, ‘projecting’, ‘round’})
- set_ec(color)¶
Alias for set_edgecolor.
- set_fc(color)¶
Alias for set_facecolor.
- set_hatch(hatch)[source]¶
Set the hatching pattern.
hatch can be one of:
/ - diagonal hatching \ - back diagonal | - vertical - - horizontal + - crossed x - crossed diagonal o - small circle O - large circle . - dots * - stars
Letters can be combined, in which case all the specified hatchings are done. If same letter repeats, it increases the density of hatching of that pattern.
Hatching is supported in the PostScript, PDF, SVG and Agg backends only.
- Parameters
hatch ({‘/’, ‘\’, ‘|’, ‘-’, ‘+’, ‘x’, ‘o’, ‘O’, ‘.’, ‘’}*)
- set_joinstyle(s)[source]¶
Set the .JoinStyle.
- Parameters
s (.JoinStyle or {‘miter’, ‘round’, ‘bevel’})
- set_linestyle(ls)[source]¶
Set the patch linestyle.
linestyle
description
'-'
or'solid'
solid line
'--'
or'dashed'
dashed line
'-.'
or'dashdot'
dash-dotted line
':'
or'dotted'
dotted line
'none'
,'None'
,' '
, or''
draw nothing
Alternatively a dash tuple of the following form can be provided:
(offset, onoffseq)
where
onoffseq
is an even length tuple of on and off ink in points.- Parameters
ls ({‘-’, ‘–’, ‘-.’, ‘:’, ‘’, (offset, on-off-seq), …}) – The line style.
- set_ls(ls)¶
Alias for set_linestyle.
- set_lw(w)¶
Alias for set_linewidth.
- class imc.types.Path(*args, **kwargs)[source]¶
A pathlib.Path child class that allows concatenation with strings by overloading the addition operator.
In addition, it implements the
startswith
andendswith
methods just like in the basestr
type.The
replace_
implementation is meant to be an implementation closer to thestr
type.Iterating over a directory with
iterdir
that does not exists will return an empty iterator instead of throwing an error.Creating a directory with
mkdir
allows existing directory and creates parents by default.- glob(pattern)[source]¶
Iterate over this subtree and yield all existing files (of any kind, including directories) matching the given relative pattern.
- Return type
- class imc.types.Series(data=None, index=None, dtype=None, name=None, copy=False, fastpath=False)[source]¶
One-dimensional ndarray with axis labels (including time series).
Labels need not be unique but must be a hashable type. The object supports both integer- and label-based indexing and provides a host of methods for performing operations involving the index. Statistical methods from ndarray have been overridden to automatically exclude missing data (currently represented as NaN).
Operations between Series (+, -, /, *, **) align values based on their associated index values– they need not be the same length. The result index will be the sorted union of the two indexes.
- Parameters
data (array-like, Iterable, dict, or scalar value) – Contains data stored in Series. If data is a dict, argument order is maintained.
index (array-like or Index (1d)) – Values must be hashable and have the same length as data. Non-unique index values are allowed. Will default to RangeIndex (0, 1, 2, …, n) if not provided. If data is dict-like and index is None, then the keys in the data are used as the index. If the index is not None, the resulting Series is reindexed with the index values.
dtype (str, numpy.dtype, or ExtensionDtype, optional) – Data type for the output Series. If not specified, this will be inferred from data. See the user guide for more usages.
name (str, optional) – The name to give to the Series.
copy (bool, default False) – Copy input data. Only affects Series or 1d ndarray input. See examples.
Examples
Constructing Series from a dictionary with an Index specified
>>> d = {'a': 1, 'b': 2, 'c': 3} >>> ser = pd.Series(data=d, index=['a', 'b', 'c']) >>> ser a 1 b 2 c 3 dtype: int64
The keys of the dictionary match with the Index values, hence the Index values have no effect.
>>> d = {'a': 1, 'b': 2, 'c': 3} >>> ser = pd.Series(data=d, index=['x', 'y', 'z']) >>> ser x NaN y NaN z NaN dtype: float64
Note that the Index is first build with the keys from the dictionary. After this the Series is reindexed with the given Index values, hence we get all NaN as a result.
Constructing Series from a list with copy=False.
>>> r = [1, 2] >>> ser = pd.Series(r, copy=False) >>> ser.iloc[0] = 999 >>> r [1, 2] >>> ser 0 999 1 2 dtype: int64
Due to input data type the Series has a copy of the original data even though copy=False, so the data is unchanged.
Constructing Series from a 1d ndarray with copy=False.
>>> r = np.array([1, 2]) >>> ser = pd.Series(r, copy=False) >>> ser.iloc[0] = 999 >>> r array([999, 2]) >>> ser 0 999 1 2 dtype: int64
Due to input data type the Series has a view on the original data, so the data is changed as well.
- add(other, level=None, fill_value=None, axis=0)¶
Return Addition of series and other, element-wise (binary operator add).
Equivalent to
series + other
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.radd
Reverse of the Addition operator, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.add(b, fill_value=0) a 2.0 b 1.0 c 1.0 d 1.0 e NaN dtype: float64
- agg(func=None, axis=0, *args, **kwargs)¶
Aggregate using one or more operations over the specified axis.
- Parameters
func (function, str, list or dict) – Function to use for aggregating the data. If a function, must either work when passed a Series or when passed to Series.apply.
Accepted combinations are:
function
string function name
list of functions and/or function names, e.g.
[np.sum, 'mean']
dict of axis labels -> functions, function names or list of such.
axis ({0 or ‘index’}) – Parameter needed for compatibility with DataFrame.
*args – Positional arguments to pass to func.
**kwargs – Keyword arguments to pass to func.
- Returns
The return can be:
scalar : when Series.agg is called with single function
Series : when DataFrame.agg is called with a single function
DataFrame : when DataFrame.agg is called with several functions
Return scalar, Series or DataFrame.
- Return type
See also
Series.apply
Invoke function on a Series.
Series.transform
Transform function producing a Series with like indexes.
Notes
agg is an alias for aggregate. Use the alias.
Functions that mutate the passed object can produce unexpected behavior or errors and are not supported. See Mutating with User Defined Function (UDF) methods for more details.
A passed user-defined-function will be passed a Series for evaluation.
Examples
>>> s = pd.Series([1, 2, 3, 4]) >>> s 0 1 1 2 2 3 3 4 dtype: int64
>>> s.agg('min') 1
>>> s.agg(['min', 'max']) min 1 max 4 dtype: int64
- aggregate(func=None, axis=0, *args, **kwargs)[source]¶
Aggregate using one or more operations over the specified axis.
- Parameters
func (function, str, list or dict) – Function to use for aggregating the data. If a function, must either work when passed a Series or when passed to Series.apply.
Accepted combinations are:
function
string function name
list of functions and/or function names, e.g.
[np.sum, 'mean']
dict of axis labels -> functions, function names or list of such.
axis ({0 or ‘index’}) – Parameter needed for compatibility with DataFrame.
*args – Positional arguments to pass to func.
**kwargs – Keyword arguments to pass to func.
- Returns
The return can be:
scalar : when Series.agg is called with single function
Series : when DataFrame.agg is called with a single function
DataFrame : when DataFrame.agg is called with several functions
Return scalar, Series or DataFrame.
- Return type
See also
Series.apply
Invoke function on a Series.
Series.transform
Transform function producing a Series with like indexes.
Notes
agg is an alias for aggregate. Use the alias.
Functions that mutate the passed object can produce unexpected behavior or errors and are not supported. See Mutating with User Defined Function (UDF) methods for more details.
A passed user-defined-function will be passed a Series for evaluation.
Examples
>>> s = pd.Series([1, 2, 3, 4]) >>> s 0 1 1 2 2 3 3 4 dtype: int64
>>> s.agg('min') 1
>>> s.agg(['min', 'max']) min 1 max 4 dtype: int64
- align(other, join='outer', axis=None, level=None, copy=True, fill_value=None, method=None, limit=None, fill_axis=0, broadcast_axis=None)[source]¶
Align two objects on their axes with the specified join method.
Join method is specified for each axis Index.
- Parameters
other (DataFrame or Series)
join ({‘outer’, ‘inner’, ‘left’, ‘right’}, default ‘outer’)
axis (allowed axis of the other object, default None) – Align on index (0), columns (1), or both (None).
level (int or level name, default None) – Broadcast across a level, matching Index values on the passed MultiIndex level.
copy (bool, default True) – Always returns new objects. If copy=False and no reindexing is required then original objects are returned.
fill_value (scalar, default np.NaN) – Value to use for missing values. Defaults to NaN, but can be any “compatible” value.
method ({‘backfill’, ‘bfill’, ‘pad’, ‘ffill’, None}, default None) – Method to use for filling holes in reindexed Series:
pad / ffill: propagate last valid observation forward to next valid.
backfill / bfill: use NEXT valid observation to fill gap.
limit (int, default None) – If method is specified, this is the maximum number of consecutive NaN values to forward/backward fill. In other words, if there is a gap with more than this number of consecutive NaNs, it will only be partially filled. If method is not specified, this is the maximum number of entries along the entire axis where NaNs will be filled. Must be greater than 0 if not None.
fill_axis ({0 or ‘index’}, default 0) – Filling axis, method and limit.
broadcast_axis ({0 or ‘index’}, default None) – Broadcast values along this axis, if aligning two objects of different dimensions.
- Returns
(left, right) – Aligned objects.
- Return type
(Series, type of other)
Examples
>>> df = pd.DataFrame( ... [[1, 2, 3, 4], [6, 7, 8, 9]], columns=["D", "B", "E", "A"], index=[1, 2] ... ) >>> other = pd.DataFrame( ... [[10, 20, 30, 40], [60, 70, 80, 90], [600, 700, 800, 900]], ... columns=["A", "B", "C", "D"], ... index=[2, 3, 4], ... ) >>> df D B E A 1 1 2 3 4 2 6 7 8 9 >>> other A B C D 2 10 20 30 40 3 60 70 80 90 4 600 700 800 900
Align on columns:
>>> left, right = df.align(other, join="outer", axis=1) >>> left A B C D E 1 4 2 NaN 1 3 2 9 7 NaN 6 8 >>> right A B C D E 2 10 20 30 40 NaN 3 60 70 80 90 NaN 4 600 700 800 900 NaN
We can also align on the index:
>>> left, right = df.align(other, join="outer", axis=0) >>> left D B E A 1 1.0 2.0 3.0 4.0 2 6.0 7.0 8.0 9.0 3 NaN NaN NaN NaN 4 NaN NaN NaN NaN >>> right A B C D 1 NaN NaN NaN NaN 2 10.0 20.0 30.0 40.0 3 60.0 70.0 80.0 90.0 4 600.0 700.0 800.0 900.0
Finally, the default axis=None will align on both index and columns:
>>> left, right = df.align(other, join="outer", axis=None) >>> left A B C D E 1 4.0 2.0 NaN 1.0 3.0 2 9.0 7.0 NaN 6.0 8.0 3 NaN NaN NaN NaN NaN 4 NaN NaN NaN NaN NaN >>> right A B C D E 1 NaN NaN NaN NaN NaN 2 10.0 20.0 30.0 40.0 NaN 3 60.0 70.0 80.0 90.0 NaN 4 600.0 700.0 800.0 900.0 NaN
- all(axis=0, bool_only=None, skipna=True, level=None, **kwargs)¶
Return whether all elements are True, potentially over an axis.
Returns True unless there at least one element within a series or along a Dataframe axis that is False or equivalent (e.g. zero or empty).
- Parameters
axis ({0 or ‘index’, 1 or ‘columns’, None}, default 0) – Indicate which axis or axes should be reduced.
0 / ‘index’ : reduce the index, return a Series whose index is the original column labels.
1 / ‘columns’ : reduce the columns, return a Series whose index is the original index.
None : reduce all axes, return a scalar.
bool_only (bool, default None) – Include only boolean columns. If None, will attempt to use everything, then use only boolean data. Not implemented for Series.
skipna (bool, default True) – Exclude NA/null values. If the entire row/column is NA and skipna is True, then the result will be True, as for an empty row/column. If skipna is False, then NA are treated as True, because these are not equal to zero.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar.
**kwargs (any, default None) – Additional keywords have no effect but might be accepted for compatibility with NumPy.
- Returns
If level is specified, then, Series is returned; otherwise, scalar is returned.
- Return type
scalar or Series
See also
Series.all
Return True if all elements are True.
DataFrame.any
Return True if one (or more) elements are True.
Examples
Series
>>> pd.Series([True, True]).all() True >>> pd.Series([True, False]).all() False >>> pd.Series([], dtype="float64").all() True >>> pd.Series([np.nan]).all() True >>> pd.Series([np.nan]).all(skipna=False) True
DataFrames
Create a dataframe from a dictionary.
>>> df = pd.DataFrame({'col1': [True, True], 'col2': [True, False]}) >>> df col1 col2 0 True True 1 True False
Default behaviour checks if column-wise values all return True.
>>> df.all() col1 True col2 False dtype: bool
Specify
axis='columns'
to check if row-wise values all return True.>>> df.all(axis='columns') 0 True 1 False dtype: bool
Or
axis=None
for whether every value is True.>>> df.all(axis=None) False
- any(axis=0, bool_only=None, skipna=True, level=None, **kwargs)¶
Return whether any element is True, potentially over an axis.
Returns False unless there is at least one element within a series or along a Dataframe axis that is True or equivalent (e.g. non-zero or non-empty).
- Parameters
axis ({0 or ‘index’, 1 or ‘columns’, None}, default 0) – Indicate which axis or axes should be reduced.
0 / ‘index’ : reduce the index, return a Series whose index is the original column labels.
1 / ‘columns’ : reduce the columns, return a Series whose index is the original index.
None : reduce all axes, return a scalar.
bool_only (bool, default None) – Include only boolean columns. If None, will attempt to use everything, then use only boolean data. Not implemented for Series.
skipna (bool, default True) – Exclude NA/null values. If the entire row/column is NA and skipna is True, then the result will be False, as for an empty row/column. If skipna is False, then NA are treated as True, because these are not equal to zero.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar.
**kwargs (any, default None) – Additional keywords have no effect but might be accepted for compatibility with NumPy.
- Returns
If level is specified, then, Series is returned; otherwise, scalar is returned.
- Return type
scalar or Series
See also
numpy.any
Numpy version of this method.
Series.any
Return whether any element is True.
Series.all
Return whether all elements are True.
DataFrame.any
Return whether any element is True over requested axis.
DataFrame.all
Return whether all elements are True over requested axis.
Examples
Series
For Series input, the output is a scalar indicating whether any element is True.
>>> pd.Series([False, False]).any() False >>> pd.Series([True, False]).any() True >>> pd.Series([], dtype="float64").any() False >>> pd.Series([np.nan]).any() False >>> pd.Series([np.nan]).any(skipna=False) True
DataFrame
Whether each column contains at least one True element (the default).
>>> df = pd.DataFrame({"A": [1, 2], "B": [0, 2], "C": [0, 0]}) >>> df A B C 0 1 0 0 1 2 2 0
>>> df.any() A True B True C False dtype: bool
Aggregating over the columns.
>>> df = pd.DataFrame({"A": [True, False], "B": [1, 2]}) >>> df A B 0 True 1 1 False 2
>>> df.any(axis='columns') 0 True 1 True dtype: bool
>>> df = pd.DataFrame({"A": [True, False], "B": [1, 0]}) >>> df A B 0 True 1 1 False 0
>>> df.any(axis='columns') 0 True 1 False dtype: bool
Aggregating over the entire DataFrame with
axis=None
.>>> df.any(axis=None) True
any for an empty DataFrame is an empty Series.
>>> pd.DataFrame([]).any() Series([], dtype: bool)
- append(to_append, ignore_index=False, verify_integrity=False)[source]¶
Concatenate two or more Series.
Deprecated since version 1.4.0: Use
concat()
instead. For further details see Deprecated DataFrame.append and Series.append- Parameters
to_append (Series or list/tuple of Series) – Series to append with self.
ignore_index (bool, default False) – If True, the resulting axis will be labeled 0, 1, …, n - 1.
verify_integrity (bool, default False) – If True, raise Exception on creating index with duplicates.
- Returns
Concatenated Series.
- Return type
See also
concat
General function to concatenate DataFrame or Series objects.
Notes
Iteratively appending to a Series can be more computationally intensive than a single concatenate. A better solution is to append values to a list and then concatenate the list with the original Series all at once.
Examples
>>> s1 = pd.Series([1, 2, 3]) >>> s2 = pd.Series([4, 5, 6]) >>> s3 = pd.Series([4, 5, 6], index=[3, 4, 5]) >>> s1.append(s2) 0 1 1 2 2 3 0 4 1 5 2 6 dtype: int64
>>> s1.append(s3) 0 1 1 2 2 3 3 4 4 5 5 6 dtype: int64
With ignore_index set to True:
>>> s1.append(s2, ignore_index=True) 0 1 1 2 2 3 3 4 4 5 5 6 dtype: int64
With verify_integrity set to True:
>>> s1.append(s2, verify_integrity=True) Traceback (most recent call last): ... ValueError: Indexes have overlapping values: [0, 1, 2]
- apply(func, convert_dtype=True, args=(), **kwargs)[source]¶
Invoke function on values of Series.
Can be ufunc (a NumPy function that applies to the entire Series) or a Python function that only works on single values.
- Parameters
func (function) – Python function or NumPy ufunc to apply.
convert_dtype (bool, default True) – Try to find better dtype for elementwise function results. If False, leave as dtype=object. Note that the dtype is always preserved for some extension array dtypes, such as Categorical.
args (tuple) – Positional arguments passed to func after the series value.
**kwargs – Additional keyword arguments passed to func.
- Returns
If func returns a Series object the result will be a DataFrame.
- Return type
See also
Series.map
For element-wise operations.
Series.agg
Only perform aggregating type operations.
Series.transform
Only perform transforming type operations.
Notes
Functions that mutate the passed object can produce unexpected behavior or errors and are not supported. See Mutating with User Defined Function (UDF) methods for more details.
Examples
Create a series with typical summer temperatures for each city.
>>> s = pd.Series([20, 21, 12], ... index=['London', 'New York', 'Helsinki']) >>> s London 20 New York 21 Helsinki 12 dtype: int64
Square the values by defining a function and passing it as an argument to
apply()
.>>> def square(x): ... return x ** 2 >>> s.apply(square) London 400 New York 441 Helsinki 144 dtype: int64
Square the values by passing an anonymous function as an argument to
apply()
.>>> s.apply(lambda x: x ** 2) London 400 New York 441 Helsinki 144 dtype: int64
Define a custom function that needs additional positional arguments and pass these additional arguments using the
args
keyword.>>> def subtract_custom_value(x, custom_value): ... return x - custom_value
>>> s.apply(subtract_custom_value, args=(5,)) London 15 New York 16 Helsinki 7 dtype: int64
Define a custom function that takes keyword arguments and pass these arguments to
apply
.>>> def add_custom_values(x, **kwargs): ... for month in kwargs: ... x += kwargs[month] ... return x
>>> s.apply(add_custom_values, june=30, july=20, august=25) London 95 New York 96 Helsinki 87 dtype: int64
Use a function from the Numpy library.
>>> s.apply(np.log) London 2.995732 New York 3.044522 Helsinki 2.484907 dtype: float64
- argsort(axis=0, kind='quicksort', order=None)[source]¶
Return the integer indices that would sort the Series values.
Override ndarray.argsort. Argsorts the value, omitting NA/null values, and places the result in the same locations as the non-NA values.
- Parameters
axis ({0 or “index”}) – Has no effect but is accepted for compatibility with numpy.
kind ({‘mergesort’, ‘quicksort’, ‘heapsort’, ‘stable’}, default ‘quicksort’) – Choice of sorting algorithm. See
numpy.sort()
for more information. ‘mergesort’ and ‘stable’ are the only stable algorithms.order (None) – Has no effect but is accepted for compatibility with numpy.
- Returns
Positions of values within the sort order with -1 indicating nan values.
- Return type
Series[np.intp]
See also
numpy.ndarray.argsort
Returns the indices that would sort this array.
- property array: pandas.core.arrays.base.ExtensionArray¶
The ExtensionArray of the data backing this Series or Index.
- Returns
An ExtensionArray of the values stored within. For extension types, this is the actual array. For NumPy native types, this is a thin (no copy) wrapper around
numpy.ndarray
..array
differs.values
which may require converting the data to a different form.- Return type
ExtensionArray
See also
Index.to_numpy
Similar method that always returns a NumPy array.
Series.to_numpy
Similar method that always returns a NumPy array.
Notes
This table lays out the different array types for each extension dtype within pandas.
dtype
array type
category
Categorical
period
PeriodArray
interval
IntervalArray
IntegerNA
IntegerArray
string
StringArray
boolean
BooleanArray
datetime64[ns, tz]
DatetimeArray
For any 3rd-party extension types, the array type will be an ExtensionArray.
For all remaining dtypes
.array
will be aarrays.NumpyExtensionArray
wrapping the actual ndarray stored within. If you absolutely need a NumPy array (possibly with copying / coercing data), then useSeries.to_numpy()
instead.Examples
For regular NumPy types like int, and float, a PandasArray is returned.
>>> pd.Series([1, 2, 3]).array <PandasArray> [1, 2, 3] Length: 3, dtype: int64
For extension types, like Categorical, the actual ExtensionArray is returned
>>> ser = pd.Series(pd.Categorical(['a', 'b', 'a'])) >>> ser.array ['a', 'b', 'a'] Categories (2, object): ['a', 'b']
- asfreq(freq, method=None, how=None, normalize=False, fill_value=None)[source]¶
Convert time series to specified frequency.
Returns the original data conformed to a new index with the specified frequency.
If the index of this Series is a
PeriodIndex
, the new index is the result of transforming the original index withPeriodIndex.asfreq
(so the original index will map one-to-one to the new index).Otherwise, the new index will be equivalent to
pd.date_range(start, end, freq=freq)
wherestart
andend
are, respectively, the first and last entries in the original index (seepandas.date_range()
). The values corresponding to any timesteps in the new index which were not present in the original index will be null (NaN
), unless a method for filling such unknowns is provided (see themethod
parameter below).The
resample()
method is more appropriate if an operation on each group of timesteps (such as an aggregate) is necessary to represent the data at the new frequency.- Parameters
freq (DateOffset or str) – Frequency DateOffset or string.
method ({‘backfill’/’bfill’, ‘pad’/’ffill’}, default None) – Method to use for filling holes in reindexed Series (note this does not fill NaNs that already were present):
‘pad’ / ‘ffill’: propagate last valid observation forward to next valid
‘backfill’ / ‘bfill’: use NEXT valid observation to fill.
how ({‘start’, ‘end’}, default end) – For PeriodIndex only (see PeriodIndex.asfreq).
normalize (bool, default False) – Whether to reset output index to midnight.
fill_value (scalar, optional) – Value to use for missing values, applied during upsampling (note this does not fill NaNs that already were present).
- Returns
Series object reindexed to the specified frequency.
- Return type
See also
reindex
Conform DataFrame to new index with optional filling logic.
Notes
To learn more about the frequency strings, please see this link.
Examples
Start by creating a series with 4 one minute timestamps.
>>> index = pd.date_range('1/1/2000', periods=4, freq='T') >>> series = pd.Series([0.0, None, 2.0, 3.0], index=index) >>> df = pd.DataFrame({'s': series}) >>> df s 2000-01-01 00:00:00 0.0 2000-01-01 00:01:00 NaN 2000-01-01 00:02:00 2.0 2000-01-01 00:03:00 3.0
Upsample the series into 30 second bins.
>>> df.asfreq(freq='30S') s 2000-01-01 00:00:00 0.0 2000-01-01 00:00:30 NaN 2000-01-01 00:01:00 NaN 2000-01-01 00:01:30 NaN 2000-01-01 00:02:00 2.0 2000-01-01 00:02:30 NaN 2000-01-01 00:03:00 3.0
Upsample again, providing a
fill value
.>>> df.asfreq(freq='30S', fill_value=9.0) s 2000-01-01 00:00:00 0.0 2000-01-01 00:00:30 9.0 2000-01-01 00:01:00 NaN 2000-01-01 00:01:30 9.0 2000-01-01 00:02:00 2.0 2000-01-01 00:02:30 9.0 2000-01-01 00:03:00 3.0
Upsample again, providing a
method
.>>> df.asfreq(freq='30S', method='bfill') s 2000-01-01 00:00:00 0.0 2000-01-01 00:00:30 NaN 2000-01-01 00:01:00 NaN 2000-01-01 00:01:30 2.0 2000-01-01 00:02:00 2.0 2000-01-01 00:02:30 3.0 2000-01-01 00:03:00 3.0
- autocorr(lag=1)[source]¶
Compute the lag-N autocorrelation.
This method computes the Pearson correlation between the Series and its shifted self.
- Parameters
lag (int, default 1) – Number of lags to apply before performing autocorrelation.
- Returns
The Pearson correlation between self and self.shift(lag).
- Return type
See also
Series.corr
Compute the correlation between two Series.
Series.shift
Shift index by desired number of periods.
DataFrame.corr
Compute pairwise correlation of columns.
DataFrame.corrwith
Compute pairwise correlation between rows or columns of two DataFrame objects.
Notes
If the Pearson correlation is not well defined return ‘NaN’.
Examples
>>> s = pd.Series([0.25, 0.5, 0.2, -0.05]) >>> s.autocorr() 0.10355... >>> s.autocorr(lag=2) -0.99999...
If the Pearson correlation is not well defined, then ‘NaN’ is returned.
>>> s = pd.Series([1, 0, 0, 0]) >>> s.autocorr() nan
- property axes: list[pandas.core.indexes.base.Index]¶
Return a list of the row axis labels.
- between(left, right, inclusive='both')[source]¶
Return boolean Series equivalent to left <= series <= right.
This function returns a boolean vector containing True wherever the corresponding Series element is between the boundary values left and right. NA values are treated as False.
- Parameters
left (scalar or list-like) – Left boundary.
right (scalar or list-like) – Right boundary.
inclusive ({“both”, “neither”, “left”, “right”}) – Include boundaries. Whether to set each bound as closed or open.
Changed in version 1.3.0.
- Returns
Series representing whether each element is between left and right (inclusive).
- Return type
Notes
This function is equivalent to
(left <= ser) & (ser <= right)
Examples
>>> s = pd.Series([2, 0, 4, 8, np.nan])
Boundary values are included by default:
>>> s.between(1, 4) 0 True 1 False 2 True 3 False 4 False dtype: bool
With inclusive set to
"neither"
boundary values are excluded:>>> s.between(1, 4, inclusive="neither") 0 True 1 False 2 False 3 False 4 False dtype: bool
left and right can be any scalar value:
>>> s = pd.Series(['Alice', 'Bob', 'Carol', 'Eve']) >>> s.between('Anna', 'Daniel') 0 False 1 True 2 True 3 False dtype: bool
- bfill(axis=None, inplace=False, limit=None, downcast=None)[source]¶
Synonym for
DataFrame.fillna()
withmethod='bfill'
.- Returns
Object with missing values filled or None if
inplace=True
.- Return type
Series/DataFrame or None
- cat¶
alias of
pandas.core.arrays.categorical.CategoricalAccessor
- clip(lower=None, upper=None, axis=None, inplace=False, *args, **kwargs)[source]¶
Trim values at input threshold(s).
Assigns values outside boundary to boundary values. Thresholds can be singular values or array like, and in the latter case the clipping is performed element-wise in the specified axis.
- Parameters
lower (float or array-like, default None) – Minimum threshold value. All values below this threshold will be set to it. A missing threshold (e.g NA) will not clip the value.
upper (float or array-like, default None) – Maximum threshold value. All values above this threshold will be set to it. A missing threshold (e.g NA) will not clip the value.
axis (int or str axis name, optional) – Align object with lower and upper along the given axis.
inplace (bool, default False) – Whether to perform the operation in place on the data.
*args, **kwargs – Additional keywords have no effect but might be accepted for compatibility with numpy.
- Returns
Same type as calling object with the values outside the clip boundaries replaced or None if
inplace=True
.- Return type
See also
Series.clip
Trim values at input threshold in series.
DataFrame.clip
Trim values at input threshold in dataframe.
numpy.clip
Clip (limit) the values in an array.
Examples
>>> data = {'col_0': [9, -3, 0, -1, 5], 'col_1': [-2, -7, 6, 8, -5]} >>> df = pd.DataFrame(data) >>> df col_0 col_1 0 9 -2 1 -3 -7 2 0 6 3 -1 8 4 5 -5
Clips per column using lower and upper thresholds:
>>> df.clip(-4, 6) col_0 col_1 0 6 -2 1 -3 -4 2 0 6 3 -1 6 4 5 -4
Clips using specific lower and upper thresholds per column element:
>>> t = pd.Series([2, -4, -1, 6, 3]) >>> t 0 2 1 -4 2 -1 3 6 4 3 dtype: int64
>>> df.clip(t, t + 4, axis=0) col_0 col_1 0 6 2 1 -3 -4 2 0 3 3 6 8 4 5 3
Clips using specific lower threshold per column element, with missing values:
>>> t = pd.Series([2, -4, np.NaN, 6, 3]) >>> t 0 2.0 1 -4.0 2 NaN 3 6.0 4 3.0 dtype: float64
>>> df.clip(t, axis=0) col_0 col_1 0 9 2 1 -3 -4 2 0 6 3 6 8 4 5 3
- combine(other, func, fill_value=None)[source]¶
Combine the Series with a Series or scalar according to func.
Combine the Series and other using func to perform elementwise selection for combined Series. fill_value is assumed when value is missing at some index from one of the two objects being combined.
- Parameters
other (Series or scalar) – The value(s) to be combined with the Series.
func (function) – Function that takes two scalars as inputs and returns an element.
fill_value (scalar, optional) – The value to assume when an index is missing from one Series or the other. The default specifies to use the appropriate NaN value for the underlying dtype of the Series.
- Returns
The result of combining the Series with the other object.
- Return type
See also
Series.combine_first
Combine Series values, choosing the calling Series’ values first.
Examples
Consider 2 Datasets
s1
ands2
containing highest clocked speeds of different birds.>>> s1 = pd.Series({'falcon': 330.0, 'eagle': 160.0}) >>> s1 falcon 330.0 eagle 160.0 dtype: float64 >>> s2 = pd.Series({'falcon': 345.0, 'eagle': 200.0, 'duck': 30.0}) >>> s2 falcon 345.0 eagle 200.0 duck 30.0 dtype: float64
Now, to combine the two datasets and view the highest speeds of the birds across the two datasets
>>> s1.combine(s2, max) duck NaN eagle 200.0 falcon 345.0 dtype: float64
In the previous example, the resulting value for duck is missing, because the maximum of a NaN and a float is a NaN. So, in the example, we set
fill_value=0
, so the maximum value returned will be the value from some dataset.>>> s1.combine(s2, max, fill_value=0) duck 30.0 eagle 200.0 falcon 345.0 dtype: float64
- combine_first(other)[source]¶
Update null elements with value in the same location in ‘other’.
Combine two Series objects by filling null values in one Series with non-null values from the other Series. Result index will be the union of the two indexes.
- Parameters
other (Series) – The value(s) to be used for filling null values.
- Returns
The result of combining the provided Series with the other object.
- Return type
See also
Series.combine
Perform element-wise operation on two Series using a given function.
Examples
>>> s1 = pd.Series([1, np.nan]) >>> s2 = pd.Series([3, 4, 5]) >>> s1.combine_first(s2) 0 1.0 1 4.0 2 5.0 dtype: float64
Null values still persist if the location of that null value does not exist in other
>>> s1 = pd.Series({'falcon': np.nan, 'eagle': 160.0}) >>> s2 = pd.Series({'eagle': 200.0, 'duck': 30.0}) >>> s1.combine_first(s2) duck 30.0 eagle 160.0 falcon NaN dtype: float64
- compare(other, align_axis=1, keep_shape=False, keep_equal=False)[source]¶
Compare to another Series and show the differences.
New in version 1.1.0.
- Parameters
other (Series) – Object to compare with.
align_axis ({0 or ‘index’, 1 or ‘columns’}, default 1) – Determine which axis to align the comparison on.
- 0, or ‘index’Resulting differences are stacked vertically
with rows drawn alternately from self and other.
- 1, or ‘columns’Resulting differences are aligned horizontally
with columns drawn alternately from self and other.
keep_shape (bool, default False) – If true, all rows and columns are kept. Otherwise, only the ones with different values are kept.
keep_equal (bool, default False) – If true, the result keeps values that are equal. Otherwise, equal values are shown as NaNs.
- Returns
If axis is 0 or ‘index’ the result will be a Series. The resulting index will be a MultiIndex with ‘self’ and ‘other’ stacked alternately at the inner level.
If axis is 1 or ‘columns’ the result will be a DataFrame. It will have two columns namely ‘self’ and ‘other’.
- Return type
See also
DataFrame.compare
Compare with another DataFrame and show differences.
Notes
Matching NaNs will not appear as a difference.
Examples
>>> s1 = pd.Series(["a", "b", "c", "d", "e"]) >>> s2 = pd.Series(["a", "a", "c", "b", "e"])
Align the differences on columns
>>> s1.compare(s2) self other 1 b a 3 d b
Stack the differences on indices
>>> s1.compare(s2, align_axis=0) 1 self b other a 3 self d other b dtype: object
Keep all original rows
>>> s1.compare(s2, keep_shape=True) self other 0 NaN NaN 1 b a 2 NaN NaN 3 d b 4 NaN NaN
Keep all original rows and also all original values
>>> s1.compare(s2, keep_shape=True, keep_equal=True) self other 0 a a 1 b a 2 c c 3 d b 4 e e
- corr(other, method='pearson', min_periods=None)[source]¶
Compute correlation with other Series, excluding missing values.
- Parameters
other (Series) – Series with which to compute the correlation.
method ({‘pearson’, ‘kendall’, ‘spearman’} or callable) – Method used to compute correlation:
pearson : Standard correlation coefficient
kendall : Kendall Tau correlation coefficient
spearman : Spearman rank correlation
callable: Callable with input two 1d ndarrays and returning a float.
Warning
Note that the returned matrix from corr will have 1 along the diagonals and will be symmetric regardless of the callable’s behavior.
min_periods (int, optional) – Minimum number of observations needed to have a valid result.
- Returns
Correlation with other.
- Return type
See also
DataFrame.corr
Compute pairwise correlation between columns.
DataFrame.corrwith
Compute pairwise correlation with another DataFrame or Series.
Examples
>>> def histogram_intersection(a, b): ... v = np.minimum(a, b).sum().round(decimals=1) ... return v >>> s1 = pd.Series([.2, .0, .6, .2]) >>> s2 = pd.Series([.3, .6, .0, .1]) >>> s1.corr(s2, method=histogram_intersection) 0.3
- count(level=None)[source]¶
Return number of non-NA/null observations in the Series.
- Parameters
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a smaller Series.
- Returns
Number of non-null values in the Series.
- Return type
See also
DataFrame.count
Count non-NA cells for each column or row.
Examples
>>> s = pd.Series([0.0, 1.0, np.nan]) >>> s.count() 2
- cov(other, min_periods=None, ddof=1)[source]¶
Compute covariance with Series, excluding missing values.
- Parameters
other (Series) – Series with which to compute the covariance.
min_periods (int, optional) – Minimum number of observations needed to have a valid result.
ddof (int, default 1) – Delta degrees of freedom. The divisor used in calculations is
N - ddof
, whereN
represents the number of elements.New in version 1.1.0.
- Returns
Covariance between Series and other normalized by N-1 (unbiased estimator).
- Return type
See also
DataFrame.cov
Compute pairwise covariance of columns.
Examples
>>> s1 = pd.Series([0.90010907, 0.13484424, 0.62036035]) >>> s2 = pd.Series([0.12528585, 0.26962463, 0.51111198]) >>> s1.cov(s2) -0.01685762652715874
- cummax(axis=None, skipna=True, *args, **kwargs)¶
Return cumulative maximum over a DataFrame or Series axis.
Returns a DataFrame or Series of the same size containing the cumulative maximum.
- Parameters
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – The index or the name of the axis. 0 is equivalent to None or ‘index’.
skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA.
*args, **kwargs – Additional keywords have no effect but might be accepted for compatibility with NumPy.
- Returns
Return cumulative maximum of scalar or Series.
- Return type
scalar or Series
See also
core.window.Expanding.max
Similar functionality but ignores
NaN
values.Series.max
Return the maximum over Series axis.
Series.cummax
Return cumulative maximum over Series axis.
Series.cummin
Return cumulative minimum over Series axis.
Series.cumsum
Return cumulative sum over Series axis.
Series.cumprod
Return cumulative product over Series axis.
Examples
Series
>>> s = pd.Series([2, np.nan, 5, -1, 0]) >>> s 0 2.0 1 NaN 2 5.0 3 -1.0 4 0.0 dtype: float64
By default, NA values are ignored.
>>> s.cummax() 0 2.0 1 NaN 2 5.0 3 5.0 4 5.0 dtype: float64
To include NA values in the operation, use
skipna=False
>>> s.cummax(skipna=False) 0 2.0 1 NaN 2 NaN 3 NaN 4 NaN dtype: float64
DataFrame
>>> df = pd.DataFrame([[2.0, 1.0], ... [3.0, np.nan], ... [1.0, 0.0]], ... columns=list('AB')) >>> df A B 0 2.0 1.0 1 3.0 NaN 2 1.0 0.0
By default, iterates over rows and finds the maximum in each column. This is equivalent to
axis=None
oraxis='index'
.>>> df.cummax() A B 0 2.0 1.0 1 3.0 NaN 2 3.0 1.0
To iterate over columns and find the maximum in each row, use
axis=1
>>> df.cummax(axis=1) A B 0 2.0 2.0 1 3.0 NaN 2 1.0 1.0
- cummin(axis=None, skipna=True, *args, **kwargs)¶
Return cumulative minimum over a DataFrame or Series axis.
Returns a DataFrame or Series of the same size containing the cumulative minimum.
- Parameters
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – The index or the name of the axis. 0 is equivalent to None or ‘index’.
skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA.
*args, **kwargs – Additional keywords have no effect but might be accepted for compatibility with NumPy.
- Returns
Return cumulative minimum of scalar or Series.
- Return type
scalar or Series
See also
core.window.Expanding.min
Similar functionality but ignores
NaN
values.Series.min
Return the minimum over Series axis.
Series.cummax
Return cumulative maximum over Series axis.
Series.cummin
Return cumulative minimum over Series axis.
Series.cumsum
Return cumulative sum over Series axis.
Series.cumprod
Return cumulative product over Series axis.
Examples
Series
>>> s = pd.Series([2, np.nan, 5, -1, 0]) >>> s 0 2.0 1 NaN 2 5.0 3 -1.0 4 0.0 dtype: float64
By default, NA values are ignored.
>>> s.cummin() 0 2.0 1 NaN 2 2.0 3 -1.0 4 -1.0 dtype: float64
To include NA values in the operation, use
skipna=False
>>> s.cummin(skipna=False) 0 2.0 1 NaN 2 NaN 3 NaN 4 NaN dtype: float64
DataFrame
>>> df = pd.DataFrame([[2.0, 1.0], ... [3.0, np.nan], ... [1.0, 0.0]], ... columns=list('AB')) >>> df A B 0 2.0 1.0 1 3.0 NaN 2 1.0 0.0
By default, iterates over rows and finds the minimum in each column. This is equivalent to
axis=None
oraxis='index'
.>>> df.cummin() A B 0 2.0 1.0 1 2.0 NaN 2 1.0 0.0
To iterate over columns and find the minimum in each row, use
axis=1
>>> df.cummin(axis=1) A B 0 2.0 1.0 1 3.0 NaN 2 1.0 0.0
- cumprod(axis=None, skipna=True, *args, **kwargs)¶
Return cumulative product over a DataFrame or Series axis.
Returns a DataFrame or Series of the same size containing the cumulative product.
- Parameters
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – The index or the name of the axis. 0 is equivalent to None or ‘index’.
skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA.
*args, **kwargs – Additional keywords have no effect but might be accepted for compatibility with NumPy.
- Returns
Return cumulative product of scalar or Series.
- Return type
scalar or Series
See also
core.window.Expanding.prod
Similar functionality but ignores
NaN
values.Series.prod
Return the product over Series axis.
Series.cummax
Return cumulative maximum over Series axis.
Series.cummin
Return cumulative minimum over Series axis.
Series.cumsum
Return cumulative sum over Series axis.
Series.cumprod
Return cumulative product over Series axis.
Examples
Series
>>> s = pd.Series([2, np.nan, 5, -1, 0]) >>> s 0 2.0 1 NaN 2 5.0 3 -1.0 4 0.0 dtype: float64
By default, NA values are ignored.
>>> s.cumprod() 0 2.0 1 NaN 2 10.0 3 -10.0 4 -0.0 dtype: float64
To include NA values in the operation, use
skipna=False
>>> s.cumprod(skipna=False) 0 2.0 1 NaN 2 NaN 3 NaN 4 NaN dtype: float64
DataFrame
>>> df = pd.DataFrame([[2.0, 1.0], ... [3.0, np.nan], ... [1.0, 0.0]], ... columns=list('AB')) >>> df A B 0 2.0 1.0 1 3.0 NaN 2 1.0 0.0
By default, iterates over rows and finds the product in each column. This is equivalent to
axis=None
oraxis='index'
.>>> df.cumprod() A B 0 2.0 1.0 1 6.0 NaN 2 6.0 0.0
To iterate over columns and find the product in each row, use
axis=1
>>> df.cumprod(axis=1) A B 0 2.0 2.0 1 3.0 NaN 2 1.0 0.0
- cumsum(axis=None, skipna=True, *args, **kwargs)¶
Return cumulative sum over a DataFrame or Series axis.
Returns a DataFrame or Series of the same size containing the cumulative sum.
- Parameters
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – The index or the name of the axis. 0 is equivalent to None or ‘index’.
skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA.
*args, **kwargs – Additional keywords have no effect but might be accepted for compatibility with NumPy.
- Returns
Return cumulative sum of scalar or Series.
- Return type
scalar or Series
See also
core.window.Expanding.sum
Similar functionality but ignores
NaN
values.Series.sum
Return the sum over Series axis.
Series.cummax
Return cumulative maximum over Series axis.
Series.cummin
Return cumulative minimum over Series axis.
Series.cumsum
Return cumulative sum over Series axis.
Series.cumprod
Return cumulative product over Series axis.
Examples
Series
>>> s = pd.Series([2, np.nan, 5, -1, 0]) >>> s 0 2.0 1 NaN 2 5.0 3 -1.0 4 0.0 dtype: float64
By default, NA values are ignored.
>>> s.cumsum() 0 2.0 1 NaN 2 7.0 3 6.0 4 6.0 dtype: float64
To include NA values in the operation, use
skipna=False
>>> s.cumsum(skipna=False) 0 2.0 1 NaN 2 NaN 3 NaN 4 NaN dtype: float64
DataFrame
>>> df = pd.DataFrame([[2.0, 1.0], ... [3.0, np.nan], ... [1.0, 0.0]], ... columns=list('AB')) >>> df A B 0 2.0 1.0 1 3.0 NaN 2 1.0 0.0
By default, iterates over rows and finds the sum in each column. This is equivalent to
axis=None
oraxis='index'
.>>> df.cumsum() A B 0 2.0 1.0 1 5.0 NaN 2 6.0 1.0
To iterate over columns and find the sum in each row, use
axis=1
>>> df.cumsum(axis=1) A B 0 2.0 3.0 1 3.0 NaN 2 1.0 1.0
- diff(periods=1)[source]¶
First discrete difference of element.
Calculates the difference of a Series element compared with another element in the Series (default is element in previous row).
- Parameters
periods (int, default 1) – Periods to shift for calculating difference, accepts negative values.
- Returns
First differences of the Series.
- Return type
See also
Series.pct_change
Percent change over given number of periods.
Series.shift
Shift index by desired number of periods with an optional time freq.
DataFrame.diff
First discrete difference of object.
Notes
For boolean dtypes, this uses
operator.xor()
rather thanoperator.sub()
. The result is calculated according to current dtype in Series, however dtype of the result is always float64.Examples
Difference with previous row
>>> s = pd.Series([1, 1, 2, 3, 5, 8]) >>> s.diff() 0 NaN 1 0.0 2 1.0 3 1.0 4 2.0 5 3.0 dtype: float64
Difference with 3rd previous row
>>> s.diff(periods=3) 0 NaN 1 NaN 2 NaN 3 2.0 4 4.0 5 6.0 dtype: float64
Difference with following row
>>> s.diff(periods=-1) 0 0.0 1 -1.0 2 -1.0 3 -2.0 4 -3.0 5 NaN dtype: float64
Overflow in input dtype
>>> s = pd.Series([1, 0], dtype=np.uint8) >>> s.diff() 0 NaN 1 255.0 dtype: float64
- div(other, level=None, fill_value=None, axis=0)¶
Return Floating division of series and other, element-wise (binary operator truediv).
Equivalent to
series / other
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.rtruediv
Reverse of the Floating division operator, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.divide(b, fill_value=0) a 1.0 b inf c inf d 0.0 e NaN dtype: float64
- divide(other, level=None, fill_value=None, axis=0)¶
Return Floating division of series and other, element-wise (binary operator truediv).
Equivalent to
series / other
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.rtruediv
Reverse of the Floating division operator, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.divide(b, fill_value=0) a 1.0 b inf c inf d 0.0 e NaN dtype: float64
- divmod(other, level=None, fill_value=None, axis=0)¶
Return Integer division and modulo of series and other, element-wise (binary operator divmod).
Equivalent to
divmod(series, other)
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
2-Tuple of Series
See also
Series.rdivmod
Reverse of the Integer division and modulo operator, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.divmod(b, fill_value=0) (a 1.0 b NaN c NaN d 0.0 e NaN dtype: float64, a 0.0 b NaN c NaN d 0.0 e NaN dtype: float64)
- dot(other)[source]¶
Compute the dot product between the Series and the columns of other.
This method computes the dot product between the Series and another one, or the Series and each columns of a DataFrame, or the Series and each columns of an array.
It can also be called using self @ other in Python >= 3.5.
- Parameters
other (Series, DataFrame or array-like) – The other object to compute the dot product with its columns.
- Returns
Return the dot product of the Series and other if other is a Series, the Series of the dot product of Series and each rows of other if other is a DataFrame or a numpy.ndarray between the Series and each columns of the numpy array.
- Return type
scalar, Series or numpy.ndarray
See also
DataFrame.dot
Compute the matrix product with the DataFrame.
Series.mul
Multiplication of series and other, element-wise.
Notes
The Series and other has to share the same index if other is a Series or a DataFrame.
Examples
>>> s = pd.Series([0, 1, 2, 3]) >>> other = pd.Series([-1, 2, -3, 4]) >>> s.dot(other) 8 >>> s @ other 8 >>> df = pd.DataFrame([[0, 1], [-2, 3], [4, -5], [6, 7]]) >>> s.dot(df) 0 24 1 14 dtype: int64 >>> arr = np.array([[0, 1], [-2, 3], [4, -5], [6, 7]]) >>> s.dot(arr) array([24, 14])
- drop(labels=None, axis=0, index=None, columns=None, level=None, inplace=False, errors='raise')[source]¶
Return Series with specified index labels removed.
Remove elements of a Series based on specifying the index labels. When using a multi-index, labels on different levels can be removed by specifying the level.
- Parameters
labels (single label or list-like) – Index labels to drop.
axis (0, default 0) – Redundant for application on Series.
index (single label or list-like) – Redundant for application on Series, but ‘index’ can be used instead of ‘labels’.
columns (single label or list-like) – No change is made to the Series; use ‘index’ or ‘labels’ instead.
level (int or level name, optional) – For MultiIndex, level for which the labels will be removed.
inplace (bool, default False) – If True, do operation inplace and return None.
errors ({‘ignore’, ‘raise’}, default ‘raise’) – If ‘ignore’, suppress error and only existing labels are dropped.
- Returns
Series with specified index labels removed or None if
inplace=True
.- Return type
Series or None
- Raises
KeyError – If none of the labels are found in the index.
See also
Series.reindex
Return only specified index labels of Series.
Series.dropna
Return series without null values.
Series.drop_duplicates
Return Series with duplicate values removed.
DataFrame.drop
Drop specified labels from rows or columns.
Examples
>>> s = pd.Series(data=np.arange(3), index=['A', 'B', 'C']) >>> s A 0 B 1 C 2 dtype: int64
Drop labels B en C
>>> s.drop(labels=['B', 'C']) A 0 dtype: int64
Drop 2nd level label in MultiIndex Series
>>> midx = pd.MultiIndex(levels=[['lama', 'cow', 'falcon'], ... ['speed', 'weight', 'length']], ... codes=[[0, 0, 0, 1, 1, 1, 2, 2, 2], ... [0, 1, 2, 0, 1, 2, 0, 1, 2]]) >>> s = pd.Series([45, 200, 1.2, 30, 250, 1.5, 320, 1, 0.3], ... index=midx) >>> s lama speed 45.0 weight 200.0 length 1.2 cow speed 30.0 weight 250.0 length 1.5 falcon speed 320.0 weight 1.0 length 0.3 dtype: float64
>>> s.drop(labels='weight', level=1) lama speed 45.0 length 1.2 cow speed 30.0 length 1.5 falcon speed 320.0 length 0.3 dtype: float64
- drop_duplicates(keep='first', inplace: Literal[False] = False) Series [source]¶
- drop_duplicates(keep, inplace: Literal[True]) None
- drop_duplicates(*, inplace: Literal[True]) None
- drop_duplicates(keep='first', inplace: bool = False) Series | None
Return Series with duplicate values removed.
- Parameters
keep ({‘first’, ‘last’,
False
}, default ‘first’) – Method to handle dropping duplicates:‘first’ : Drop duplicates except for the first occurrence.
‘last’ : Drop duplicates except for the last occurrence.
False
: Drop all duplicates.
inplace (bool, default
False
) – IfTrue
, performs operation inplace and returns None.
- Returns
Series with duplicates dropped or None if
inplace=True
.- Return type
Series or None
See also
Index.drop_duplicates
Equivalent method on Index.
DataFrame.drop_duplicates
Equivalent method on DataFrame.
Series.duplicated
Related method on Series, indicating duplicate Series values.
Examples
Generate a Series with duplicated entries.
>>> s = pd.Series(['lama', 'cow', 'lama', 'beetle', 'lama', 'hippo'], ... name='animal') >>> s 0 lama 1 cow 2 lama 3 beetle 4 lama 5 hippo Name: animal, dtype: object
With the ‘keep’ parameter, the selection behaviour of duplicated values can be changed. The value ‘first’ keeps the first occurrence for each set of duplicated entries. The default value of keep is ‘first’.
>>> s.drop_duplicates() 0 lama 1 cow 3 beetle 5 hippo Name: animal, dtype: object
The value ‘last’ for parameter ‘keep’ keeps the last occurrence for each set of duplicated entries.
>>> s.drop_duplicates(keep='last') 1 cow 3 beetle 4 lama 5 hippo Name: animal, dtype: object
The value
False
for parameter ‘keep’ discards all sets of duplicated entries. Setting the value of ‘inplace’ toTrue
performs the operation inplace and returnsNone
.>>> s.drop_duplicates(keep=False, inplace=True) >>> s 1 cow 3 beetle 5 hippo Name: animal, dtype: object
- dropna(axis=0, inplace=False, how=None)[source]¶
Return a new Series with missing values removed.
See the User Guide for more on which values are considered missing, and how to work with missing data.
- Parameters
axis ({0 or ‘index’}, default 0) – There is only one axis to drop values from.
inplace (bool, default False) – If True, do operation inplace and return None.
how (str, optional) – Not in use. Kept for compatibility.
- Returns
Series with NA entries dropped from it or None if
inplace=True
.- Return type
Series or None
See also
Series.isna
Indicate missing values.
Series.notna
Indicate existing (non-missing) values.
Series.fillna
Replace missing values.
DataFrame.dropna
Drop rows or columns which contain NA values.
Index.dropna
Drop missing indices.
Examples
>>> ser = pd.Series([1., 2., np.nan]) >>> ser 0 1.0 1 2.0 2 NaN dtype: float64
Drop NA values from a Series.
>>> ser.dropna() 0 1.0 1 2.0 dtype: float64
Keep the Series with valid entries in the same variable.
>>> ser.dropna(inplace=True) >>> ser 0 1.0 1 2.0 dtype: float64
Empty strings are not considered NA values.
None
is considered an NA value.>>> ser = pd.Series([np.NaN, 2, pd.NaT, '', None, 'I stay']) >>> ser 0 NaN 1 2 2 NaT 3 4 None 5 I stay dtype: object >>> ser.dropna() 1 2 3 5 I stay dtype: object
- dt¶
alias of
pandas.core.indexes.accessors.CombinedDatetimelikeProperties
- property dtype: DtypeObj¶
Return the dtype object of the underlying data.
- Return type
DtypeObj
- property dtypes: DtypeObj¶
Return the dtype object of the underlying data.
- Return type
DtypeObj
- duplicated(keep='first')[source]¶
Indicate duplicate Series values.
Duplicated values are indicated as
True
values in the resulting Series. Either all duplicates, all except the first or all except the last occurrence of duplicates can be indicated.- Parameters
keep ({‘first’, ‘last’, False}, default ‘first’) – Method to handle dropping duplicates:
‘first’ : Mark duplicates as
True
except for the first occurrence.‘last’ : Mark duplicates as
True
except for the last occurrence.False
: Mark all duplicates asTrue
.
- Returns
Series indicating whether each value has occurred in the preceding values.
- Return type
See also
Index.duplicated
Equivalent method on pandas.Index.
DataFrame.duplicated
Equivalent method on pandas.DataFrame.
Series.drop_duplicates
Remove duplicate values from Series.
Examples
By default, for each set of duplicated values, the first occurrence is set on False and all others on True:
>>> animals = pd.Series(['lama', 'cow', 'lama', 'beetle', 'lama']) >>> animals.duplicated() 0 False 1 False 2 True 3 False 4 True dtype: bool
which is equivalent to
>>> animals.duplicated(keep='first') 0 False 1 False 2 True 3 False 4 True dtype: bool
By using ‘last’, the last occurrence of each set of duplicated values is set on False and all others on True:
>>> animals.duplicated(keep='last') 0 True 1 False 2 True 3 False 4 False dtype: bool
By setting keep on
False
, all duplicates are True:>>> animals.duplicated(keep=False) 0 True 1 False 2 True 3 False 4 True dtype: bool
- eq(other, level=None, fill_value=None, axis=0)¶
Return Equal to of series and other, element-wise (binary operator eq).
Equivalent to
series == other
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.eq(b, fill_value=0) a True b False c False d False e False dtype: bool
- explode(ignore_index=False)[source]¶
Transform each element of a list-like to a row.
New in version 0.25.0.
- Parameters
ignore_index (bool, default False) – If True, the resulting index will be labeled 0, 1, …, n - 1.
New in version 1.1.0.
- Returns
Exploded lists to rows; index will be duplicated for these rows.
- Return type
See also
Series.str.split
Split string values on specified separator.
Series.unstack
Unstack, a.k.a. pivot, Series with MultiIndex to produce DataFrame.
DataFrame.melt
Unpivot a DataFrame from wide format to long format.
DataFrame.explode
Explode a DataFrame from list-like columns to long format.
Notes
This routine will explode list-likes including lists, tuples, sets, Series, and np.ndarray. The result dtype of the subset rows will be object. Scalars will be returned unchanged, and empty list-likes will result in a np.nan for that row. In addition, the ordering of elements in the output will be non-deterministic when exploding sets.
Reference the user guide for more examples.
Examples
>>> s = pd.Series([[1, 2, 3], 'foo', [], [3, 4]]) >>> s 0 [1, 2, 3] 1 foo 2 [] 3 [3, 4] dtype: object
>>> s.explode() 0 1 0 2 0 3 1 foo 2 NaN 3 3 3 4 dtype: object
- ffill(axis=None, inplace=False, limit=None, downcast=None)[source]¶
Synonym for
DataFrame.fillna()
withmethod='ffill'
.- Returns
Object with missing values filled or None if
inplace=True
.- Return type
Series/DataFrame or None
- fillna(value=None, method: FillnaOptions | None = None, axis: Axis | None = None, inplace: Literal[False] = False, limit=None, downcast=None) Series [source]¶
- fillna(value, method: FillnaOptions | None, axis: Axis | None, inplace: Literal[True], limit=None, downcast=None) None
- fillna(*, inplace: Literal[True], limit=None, downcast=None) None
- fillna(value, *, inplace: Literal[True], limit=None, downcast=None) None
- fillna(*, method: FillnaOptions | None, inplace: Literal[True], limit=None, downcast=None) None
- fillna(*, axis: Axis | None, inplace: Literal[True], limit=None, downcast=None) None
- fillna(*, method: FillnaOptions | None, axis: Axis | None, inplace: Literal[True], limit=None, downcast=None) None
- fillna(value, *, axis: Axis | None, inplace: Literal[True], limit=None, downcast=None) None
- fillna(value, method: FillnaOptions | None, *, inplace: Literal[True], limit=None, downcast=None) None
- fillna(value=None, method: FillnaOptions | None = None, axis: Axis | None = None, inplace: bool = False, limit=None, downcast=None) Series | None
Fill NA/NaN values using the specified method.
- Parameters
value (scalar, dict, Series, or DataFrame) – Value to use to fill holes (e.g. 0), alternately a dict/Series/DataFrame of values specifying which value to use for each index (for a Series) or column (for a DataFrame). Values not in the dict/Series/DataFrame will not be filled. This value cannot be a list.
method ({‘backfill’, ‘bfill’, ‘pad’, ‘ffill’, None}, default None) – Method to use for filling holes in reindexed Series pad / ffill: propagate last valid observation forward to next valid backfill / bfill: use next valid observation to fill gap.
axis ({0 or ‘index’}) – Axis along which to fill missing values.
inplace (bool, default False) – If True, fill in-place. Note: this will modify any other views on this object (e.g., a no-copy slice for a column in a DataFrame).
limit (int, default None) – If method is specified, this is the maximum number of consecutive NaN values to forward/backward fill. In other words, if there is a gap with more than this number of consecutive NaNs, it will only be partially filled. If method is not specified, this is the maximum number of entries along the entire axis where NaNs will be filled. Must be greater than 0 if not None.
downcast (dict, default is None) – A dict of item->dtype of what to downcast if possible, or the string ‘infer’ which will try to downcast to an appropriate equal type (e.g. float64 to int64 if possible).
- Returns
Object with missing values filled or None if
inplace=True
.- Return type
Series or None
See also
interpolate
Fill NaN values using interpolation.
reindex
Conform object to new index.
asfreq
Convert TimeSeries to specified frequency.
Examples
>>> df = pd.DataFrame([[np.nan, 2, np.nan, 0], ... [3, 4, np.nan, 1], ... [np.nan, np.nan, np.nan, np.nan], ... [np.nan, 3, np.nan, 4]], ... columns=list("ABCD")) >>> df A B C D 0 NaN 2.0 NaN 0.0 1 3.0 4.0 NaN 1.0 2 NaN NaN NaN NaN 3 NaN 3.0 NaN 4.0
Replace all NaN elements with 0s.
>>> df.fillna(0) A B C D 0 0.0 2.0 0.0 0.0 1 3.0 4.0 0.0 1.0 2 0.0 0.0 0.0 0.0 3 0.0 3.0 0.0 4.0
We can also propagate non-null values forward or backward.
>>> df.fillna(method="ffill") A B C D 0 NaN 2.0 NaN 0.0 1 3.0 4.0 NaN 1.0 2 3.0 4.0 NaN 1.0 3 3.0 3.0 NaN 4.0
Replace all NaN elements in column ‘A’, ‘B’, ‘C’, and ‘D’, with 0, 1, 2, and 3 respectively.
>>> values = {"A": 0, "B": 1, "C": 2, "D": 3} >>> df.fillna(value=values) A B C D 0 0.0 2.0 2.0 0.0 1 3.0 4.0 2.0 1.0 2 0.0 1.0 2.0 3.0 3 0.0 3.0 2.0 4.0
Only replace the first NaN element.
>>> df.fillna(value=values, limit=1) A B C D 0 0.0 2.0 2.0 0.0 1 3.0 4.0 NaN 1.0 2 NaN 1.0 NaN 3.0 3 NaN 3.0 NaN 4.0
When filling using a DataFrame, replacement happens along the same column names and same indices
>>> df2 = pd.DataFrame(np.zeros((4, 4)), columns=list("ABCE")) >>> df.fillna(df2) A B C D 0 0.0 2.0 0.0 0.0 1 3.0 4.0 0.0 1.0 2 0.0 0.0 0.0 NaN 3 0.0 3.0 0.0 4.0
Note that column D is not affected since it is not present in df2.
- floordiv(other, level=None, fill_value=None, axis=0)¶
Return Integer division of series and other, element-wise (binary operator floordiv).
Equivalent to
series // other
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.rfloordiv
Reverse of the Integer division operator, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.floordiv(b, fill_value=0) a 1.0 b NaN c NaN d 0.0 e NaN dtype: float64
- ge(other, level=None, fill_value=None, axis=0)¶
Return Greater than or equal to of series and other, element-wise (binary operator ge).
Equivalent to
series >= other
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
Examples
>>> a = pd.Series([1, 1, 1, np.nan, 1], index=['a', 'b', 'c', 'd', 'e']) >>> a a 1.0 b 1.0 c 1.0 d NaN e 1.0 dtype: float64 >>> b = pd.Series([0, 1, 2, np.nan, 1], index=['a', 'b', 'c', 'd', 'f']) >>> b a 0.0 b 1.0 c 2.0 d NaN f 1.0 dtype: float64 >>> a.ge(b, fill_value=0) a True b True c False d False e True f False dtype: bool
- groupby(by=None, axis=0, level=None, as_index=True, sort=True, group_keys=True, squeeze=NoDefault.no_default, observed=False, dropna=True)[source]¶
Group Series using a mapper or by a Series of columns.
A groupby operation involves some combination of splitting the object, applying a function, and combining the results. This can be used to group large amounts of data and compute operations on these groups.
- Parameters
by (mapping, function, label, or list of labels) – Used to determine the groups for the groupby. If
by
is a function, it’s called on each value of the object’s index. If a dict or Series is passed, the Series or dict VALUES will be used to determine the groups (the Series’ values are first aligned; see.align()
method). If a list or ndarray of length equal to the selected axis is passed (see the groupby user guide), the values are used as-is to determine the groups. A label or list of labels may be passed to group by the columns inself
. Notice that a tuple is interpreted as a (single) key.axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – Split along rows (0) or columns (1).
level (int, level name, or sequence of such, default None) – If the axis is a MultiIndex (hierarchical), group by a particular level or levels.
as_index (bool, default True) – For aggregated output, return object with group labels as the index. Only relevant for DataFrame input. as_index=False is effectively “SQL-style” grouped output.
sort (bool, default True) – Sort group keys. Get better performance by turning this off. Note this does not influence the order of observations within each group. Groupby preserves the order of rows within each group.
group_keys (bool, default True) – When calling apply, add group keys to index to identify pieces.
squeeze (bool, default False) – Reduce the dimensionality of the return type if possible, otherwise return a consistent type.
Deprecated since version 1.1.0.
observed (bool, default False) – This only applies if any of the groupers are Categoricals. If True: only show observed values for categorical groupers. If False: show all values for categorical groupers.
dropna (bool, default True) – If True, and if group keys contain NA values, NA values together with row/column will be dropped. If False, NA values will also be treated as the key in groups.
New in version 1.1.0.
- Returns
Returns a groupby object that contains information about the groups.
- Return type
SeriesGroupBy
See also
resample
Convenience method for frequency conversion and resampling of time series.
Notes
See the user guide for more detailed usage and examples, including splitting an object into groups, iterating through groups, selecting a group, aggregation, and more.
Examples
>>> ser = pd.Series([390., 350., 30., 20.], ... index=['Falcon', 'Falcon', 'Parrot', 'Parrot'], name="Max Speed") >>> ser Falcon 390.0 Falcon 350.0 Parrot 30.0 Parrot 20.0 Name: Max Speed, dtype: float64 >>> ser.groupby(["a", "b", "a", "b"]).mean() a 210.0 b 185.0 Name: Max Speed, dtype: float64 >>> ser.groupby(level=0).mean() Falcon 370.0 Parrot 25.0 Name: Max Speed, dtype: float64 >>> ser.groupby(ser > 100).mean() Max Speed False 25.0 True 370.0 Name: Max Speed, dtype: float64
Grouping by Indexes
We can groupby different levels of a hierarchical index using the level parameter:
>>> arrays = [['Falcon', 'Falcon', 'Parrot', 'Parrot'], ... ['Captive', 'Wild', 'Captive', 'Wild']] >>> index = pd.MultiIndex.from_arrays(arrays, names=('Animal', 'Type')) >>> ser = pd.Series([390., 350., 30., 20.], index=index, name="Max Speed") >>> ser Animal Type Falcon Captive 390.0 Wild 350.0 Parrot Captive 30.0 Wild 20.0 Name: Max Speed, dtype: float64 >>> ser.groupby(level=0).mean() Animal Falcon 370.0 Parrot 25.0 Name: Max Speed, dtype: float64 >>> ser.groupby(level="Type").mean() Type Captive 210.0 Wild 185.0 Name: Max Speed, dtype: float64
We can also choose to include NA in group keys or not by defining dropna parameter, the default setting is True.
>>> ser = pd.Series([1, 2, 3, 3], index=["a", 'a', 'b', np.nan]) >>> ser.groupby(level=0).sum() a 3 b 3 dtype: int64
>>> ser.groupby(level=0, dropna=False).sum() a 3 b 3 NaN 3 dtype: int64
>>> arrays = ['Falcon', 'Falcon', 'Parrot', 'Parrot'] >>> ser = pd.Series([390., 350., 30., 20.], index=arrays, name="Max Speed") >>> ser.groupby(["a", "b", "a", np.nan]).mean() a 210.0 b 350.0 Name: Max Speed, dtype: float64
>>> ser.groupby(["a", "b", "a", np.nan], dropna=False).mean() a 210.0 b 350.0 NaN 20.0 Name: Max Speed, dtype: float64
- gt(other, level=None, fill_value=None, axis=0)¶
Return Greater than of series and other, element-wise (binary operator gt).
Equivalent to
series > other
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
Examples
>>> a = pd.Series([1, 1, 1, np.nan, 1], index=['a', 'b', 'c', 'd', 'e']) >>> a a 1.0 b 1.0 c 1.0 d NaN e 1.0 dtype: float64 >>> b = pd.Series([0, 1, 2, np.nan, 1], index=['a', 'b', 'c', 'd', 'f']) >>> b a 0.0 b 1.0 c 2.0 d NaN f 1.0 dtype: float64 >>> a.gt(b, fill_value=0) a True b False c False d False e True f False dtype: bool
- property hasnans: bool¶
Return True if there are any NaNs.
Enables various performance speedups.
- Return type
- hist(by=None, ax=None, grid=True, xlabelsize=None, xrot=None, ylabelsize=None, yrot=None, figsize=None, bins=10, backend=None, legend=False, **kwargs)¶
Draw histogram of the input series using matplotlib.
- Parameters
by (object, optional) – If passed, then used to form histograms for separate groups.
ax (matplotlib axis object) – If not passed, uses gca().
grid (bool, default True) – Whether to show axis grid lines.
xlabelsize (int, default None) – If specified changes the x-axis label size.
xrot (float, default None) – Rotation of x axis labels.
ylabelsize (int, default None) – If specified changes the y-axis label size.
yrot (float, default None) – Rotation of y axis labels.
figsize (tuple, default None) – Figure size in inches by default.
bins (int or sequence, default 10) – Number of histogram bins to be used. If an integer is given, bins + 1 bin edges are calculated and returned. If bins is a sequence, gives bin edges, including left edge of first bin and right edge of last bin. In this case, bins is returned unmodified.
backend (str, default None) – Backend to use instead of the backend specified in the option
plotting.backend
. For instance, ‘matplotlib’. Alternatively, to specify theplotting.backend
for the whole session, setpd.options.plotting.backend
.New in version 1.0.0.
legend (bool, default False) – Whether to show the legend.
New in version 1.1.0.
**kwargs – To be passed to the actual plotting function.
- Returns
A histogram plot.
- Return type
matplotlib.AxesSubplot
See also
matplotlib.axes.Axes.hist
Plot a histogram using matplotlib.
- idxmax(axis=0, skipna=True, *args, **kwargs)[source]¶
Return the row label of the maximum value.
If multiple values equal the maximum, the first row label with that value is returned.
- Parameters
axis (int, default 0) – For compatibility with DataFrame.idxmax. Redundant for application on Series.
skipna (bool, default True) – Exclude NA/null values. If the entire Series is NA, the result will be NA.
*args, **kwargs – Additional arguments and keywords have no effect but might be accepted for compatibility with NumPy.
- Returns
Label of the maximum value.
- Return type
Index
- Raises
ValueError – If the Series is empty.
See also
numpy.argmax
Return indices of the maximum values along the given axis.
DataFrame.idxmax
Return index of first occurrence of maximum over requested axis.
Series.idxmin
Return index label of the first occurrence of minimum of values.
Notes
This method is the Series version of
ndarray.argmax
. This method returns the label of the maximum, whilendarray.argmax
returns the position. To get the position, useseries.values.argmax()
.Examples
>>> s = pd.Series(data=[1, None, 4, 3, 4], ... index=['A', 'B', 'C', 'D', 'E']) >>> s A 1.0 B NaN C 4.0 D 3.0 E 4.0 dtype: float64
>>> s.idxmax() 'C'
If skipna is False and there is an NA value in the data, the function returns
nan
.>>> s.idxmax(skipna=False) nan
- idxmin(axis=0, skipna=True, *args, **kwargs)[source]¶
Return the row label of the minimum value.
If multiple values equal the minimum, the first row label with that value is returned.
- Parameters
axis (int, default 0) – For compatibility with DataFrame.idxmin. Redundant for application on Series.
skipna (bool, default True) – Exclude NA/null values. If the entire Series is NA, the result will be NA.
*args, **kwargs – Additional arguments and keywords have no effect but might be accepted for compatibility with NumPy.
- Returns
Label of the minimum value.
- Return type
Index
- Raises
ValueError – If the Series is empty.
See also
numpy.argmin
Return indices of the minimum values along the given axis.
DataFrame.idxmin
Return index of first occurrence of minimum over requested axis.
Series.idxmax
Return index label of the first occurrence of maximum of values.
Notes
This method is the Series version of
ndarray.argmin
. This method returns the label of the minimum, whilendarray.argmin
returns the position. To get the position, useseries.values.argmin()
.Examples
>>> s = pd.Series(data=[1, None, 4, 1], ... index=['A', 'B', 'C', 'D']) >>> s A 1.0 B NaN C 4.0 D 1.0 dtype: float64
>>> s.idxmin() 'A'
If skipna is False and there is an NA value in the data, the function returns
nan
.>>> s.idxmin(skipna=False) nan
- index: Index¶
The index (axis labels) of the Series.
- info(verbose=None, buf=None, max_cols=None, memory_usage=None, show_counts=True)[source]¶
Print a concise summary of a Series.
This method prints information about a Series including the index dtype, non-null values and memory usage.
New in version 1.4.0.
- Parameters
data (Series) – Series to print information about.
verbose (bool, optional) – Whether to print the full summary. By default, the setting in
pandas.options.display.max_info_columns
is followed.buf (writable buffer, defaults to sys.stdout) – Where to send the output. By default, the output is printed to sys.stdout. Pass a writable buffer if you need to further process the output.
memory_usage (bool, str, optional) – Specifies whether total memory usage of the Series elements (including the index) should be displayed. By default, this follows the
pandas.options.display.memory_usage
setting.True always show memory usage. False never shows memory usage. A value of ‘deep’ is equivalent to “True with deep introspection”. Memory usage is shown in human-readable units (base-2 representation). Without deep introspection a memory estimation is made based in column dtype and number of rows assuming values consume the same memory amount for corresponding dtypes. With deep memory introspection, a real memory usage calculation is performed at the cost of computational resources.
show_counts (bool, optional) – Whether to show the non-null counts. By default, this is shown only if the DataFrame is smaller than
pandas.options.display.max_info_rows
andpandas.options.display.max_info_columns
. A value of True always shows the counts, and False never shows the counts.
- Returns
This method prints a summary of a Series and returns None.
- Return type
None
See also
Series.describe
Generate descriptive statistics of Series.
Series.memory_usage
Memory usage of Series.
Examples
>>> int_values = [1, 2, 3, 4, 5] >>> text_values = ['alpha', 'beta', 'gamma', 'delta', 'epsilon'] >>> s = pd.Series(text_values, index=int_values) >>> s.info() <class 'pandas.core.series.Series'> Int64Index: 5 entries, 1 to 5 Series name: None Non-Null Count Dtype -------------- ----- 5 non-null object dtypes: object(1) memory usage: 80.0+ bytes
Prints a summary excluding information about its values:
>>> s.info(verbose=False) <class 'pandas.core.series.Series'> Int64Index: 5 entries, 1 to 5 dtypes: object(1) memory usage: 80.0+ bytes
Pipe output of Series.info to buffer instead of sys.stdout, get buffer content and writes to a text file:
>>> import io >>> buffer = io.StringIO() >>> s.info(buf=buffer) >>> s = buffer.getvalue() >>> with open("df_info.txt", "w", ... encoding="utf-8") as f: ... f.write(s) 260
The memory_usage parameter allows deep introspection mode, specially useful for big Series and fine-tune memory optimization:
>>> random_strings_array = np.random.choice(['a', 'b', 'c'], 10 ** 6) >>> s = pd.Series(np.random.choice(['a', 'b', 'c'], 10 ** 6)) >>> s.info() <class 'pandas.core.series.Series'> RangeIndex: 1000000 entries, 0 to 999999 Series name: None Non-Null Count Dtype -------------- ----- 1000000 non-null object dtypes: object(1) memory usage: 7.6+ MB
>>> s.info(memory_usage='deep') <class 'pandas.core.series.Series'> RangeIndex: 1000000 entries, 0 to 999999 Series name: None Non-Null Count Dtype -------------- ----- 1000000 non-null object dtypes: object(1) memory usage: 55.3 MB
- interpolate(method='linear', axis=0, limit=None, inplace=False, limit_direction=None, limit_area=None, downcast=None, **kwargs)[source]¶
Fill NaN values using an interpolation method.
Please note that only
method='linear'
is supported for DataFrame/Series with a MultiIndex.- Parameters
method (str, default ‘linear’) – Interpolation technique to use. One of:
‘linear’: Ignore the index and treat the values as equally spaced. This is the only method supported on MultiIndexes.
‘time’: Works on daily and higher resolution data to interpolate given length of interval.
‘index’, ‘values’: use the actual numerical values of the index.
‘pad’: Fill in NaNs using existing values.
‘nearest’, ‘zero’, ‘slinear’, ‘quadratic’, ‘cubic’, ‘spline’, ‘barycentric’, ‘polynomial’: Passed to scipy.interpolate.interp1d. These methods use the numerical values of the index. Both ‘polynomial’ and ‘spline’ require that you also specify an order (int), e.g.
df.interpolate(method='polynomial', order=5)
.‘krogh’, ‘piecewise_polynomial’, ‘spline’, ‘pchip’, ‘akima’, ‘cubicspline’: Wrappers around the SciPy interpolation methods of similar names. See Notes.
‘from_derivatives’: Refers to scipy.interpolate.BPoly.from_derivatives which replaces ‘piecewise_polynomial’ interpolation method in scipy 0.18.
axis ({{0 or ‘index’, 1 or ‘columns’, None}}, default None) – Axis to interpolate along.
limit (int, optional) – Maximum number of consecutive NaNs to fill. Must be greater than 0.
inplace (bool, default False) – Update the data in place if possible.
limit_direction ({{‘forward’, ‘backward’, ‘both’}}, Optional) – Consecutive NaNs will be filled in this direction.
- If limit is specified:
If ‘method’ is ‘pad’ or ‘ffill’, ‘limit_direction’ must be ‘forward’.
If ‘method’ is ‘backfill’ or ‘bfill’, ‘limit_direction’ must be ‘backwards’.
- If ‘limit’ is not specified:
If ‘method’ is ‘backfill’ or ‘bfill’, the default is ‘backward’
else the default is ‘forward’
Changed in version 1.1.0: raises ValueError if limit_direction is ‘forward’ or ‘both’ and method is ‘backfill’ or ‘bfill’. raises ValueError if limit_direction is ‘backward’ or ‘both’ and method is ‘pad’ or ‘ffill’.
limit_area ({{None, ‘inside’, ‘outside’}}, default None) – If limit is specified, consecutive NaNs will be filled with this restriction.
None
: No fill restriction.‘inside’: Only fill NaNs surrounded by valid values (interpolate).
‘outside’: Only fill NaNs outside valid values (extrapolate).
downcast (optional, ‘infer’ or None, defaults to None) – Downcast dtypes if possible.
``**kwargs`` (optional) – Keyword arguments to pass on to the interpolating function.
- Returns
Returns the same object type as the caller, interpolated at some or all
NaN
values or None ifinplace=True
.- Return type
See also
fillna
Fill missing values using different methods.
scipy.interpolate.Akima1DInterpolator
Piecewise cubic polynomials (Akima interpolator).
scipy.interpolate.BPoly.from_derivatives
Piecewise polynomial in the Bernstein basis.
scipy.interpolate.interp1d
Interpolate a 1-D function.
scipy.interpolate.KroghInterpolator
Interpolate polynomial (Krogh interpolator).
scipy.interpolate.PchipInterpolator
PCHIP 1-d monotonic cubic interpolation.
scipy.interpolate.CubicSpline
Cubic spline data interpolator.
Notes
The ‘krogh’, ‘piecewise_polynomial’, ‘spline’, ‘pchip’ and ‘akima’ methods are wrappers around the respective SciPy implementations of similar names. These use the actual numerical values of the index. For more information on their behavior, see the SciPy documentation and SciPy tutorial.
Examples
Filling in
NaN
in aSeries
via linear interpolation.>>> s = pd.Series([0, 1, np.nan, 3]) >>> s 0 0.0 1 1.0 2 NaN 3 3.0 dtype: float64 >>> s.interpolate() 0 0.0 1 1.0 2 2.0 3 3.0 dtype: float64
Filling in
NaN
in a Series by padding, but filling at most two consecutiveNaN
at a time.>>> s = pd.Series([np.nan, "single_one", np.nan, ... "fill_two_more", np.nan, np.nan, np.nan, ... 4.71, np.nan]) >>> s 0 NaN 1 single_one 2 NaN 3 fill_two_more 4 NaN 5 NaN 6 NaN 7 4.71 8 NaN dtype: object >>> s.interpolate(method='pad', limit=2) 0 NaN 1 single_one 2 single_one 3 fill_two_more 4 fill_two_more 5 fill_two_more 6 NaN 7 4.71 8 4.71 dtype: object
Filling in
NaN
in a Series via polynomial interpolation or splines: Both ‘polynomial’ and ‘spline’ methods require that you also specify anorder
(int).>>> s = pd.Series([0, 2, np.nan, 8]) >>> s.interpolate(method='polynomial', order=2) 0 0.000000 1 2.000000 2 4.666667 3 8.000000 dtype: float64
Fill the DataFrame forward (that is, going down) along each column using linear interpolation.
Note how the last entry in column ‘a’ is interpolated differently, because there is no entry after it to use for interpolation. Note how the first entry in column ‘b’ remains
NaN
, because there is no entry before it to use for interpolation.>>> df = pd.DataFrame([(0.0, np.nan, -1.0, 1.0), ... (np.nan, 2.0, np.nan, np.nan), ... (2.0, 3.0, np.nan, 9.0), ... (np.nan, 4.0, -4.0, 16.0)], ... columns=list('abcd')) >>> df a b c d 0 0.0 NaN -1.0 1.0 1 NaN 2.0 NaN NaN 2 2.0 3.0 NaN 9.0 3 NaN 4.0 -4.0 16.0 >>> df.interpolate(method='linear', limit_direction='forward', axis=0) a b c d 0 0.0 NaN -1.0 1.0 1 1.0 2.0 -2.0 5.0 2 2.0 3.0 -3.0 9.0 3 2.0 4.0 -4.0 16.0
Using polynomial interpolation.
>>> df['d'].interpolate(method='polynomial', order=2) 0 1.0 1 4.0 2 9.0 3 16.0 Name: d, dtype: float64
- isin(values)[source]¶
Whether elements in Series are contained in values.
Return a boolean Series showing whether each element in the Series matches an element in the passed sequence of values exactly.
- Parameters
values (set or list-like) – The sequence of values to test. Passing in a single string will raise a
TypeError
. Instead, turn a single string into a list of one element.- Returns
Series of booleans indicating if each element is in values.
- Return type
- Raises
If values is a string
See also
DataFrame.isin
Equivalent method on DataFrame.
Examples
>>> s = pd.Series(['lama', 'cow', 'lama', 'beetle', 'lama', ... 'hippo'], name='animal') >>> s.isin(['cow', 'lama']) 0 True 1 True 2 True 3 False 4 True 5 False Name: animal, dtype: bool
To invert the boolean values, use the
~
operator:>>> ~s.isin(['cow', 'lama']) 0 False 1 False 2 False 3 True 4 False 5 True Name: animal, dtype: bool
Passing a single string as
s.isin('lama')
will raise an error. Use a list of one element instead:>>> s.isin(['lama']) 0 True 1 False 2 True 3 False 4 True 5 False Name: animal, dtype: bool
Strings and integers are distinct and are therefore not comparable:
>>> pd.Series([1]).isin(['1']) 0 False dtype: bool >>> pd.Series([1.1]).isin(['1.1']) 0 False dtype: bool
- isna()[source]¶
Detect missing values.
Return a boolean same-sized object indicating if the values are NA. NA values, such as None or
numpy.NaN
, gets mapped to True values. Everything else gets mapped to False values. Characters such as empty strings''
ornumpy.inf
are not considered NA values (unless you setpandas.options.mode.use_inf_as_na = True
).- Returns
Mask of bool values for each element in Series that indicates whether an element is an NA value.
- Return type
See also
Series.isnull
Alias of isna.
Series.notna
Boolean inverse of isna.
Series.dropna
Omit axes labels with missing values.
isna
Top-level isna.
Examples
Show which entries in a DataFrame are NA.
>>> df = pd.DataFrame(dict(age=[5, 6, np.NaN], ... born=[pd.NaT, pd.Timestamp('1939-05-27'), ... pd.Timestamp('1940-04-25')], ... name=['Alfred', 'Batman', ''], ... toy=[None, 'Batmobile', 'Joker'])) >>> df age born name toy 0 5.0 NaT Alfred None 1 6.0 1939-05-27 Batman Batmobile 2 NaN 1940-04-25 Joker
>>> df.isna() age born name toy 0 False True False True 1 False False False False 2 True False False False
Show which entries in a Series are NA.
>>> ser = pd.Series([5, 6, np.NaN]) >>> ser 0 5.0 1 6.0 2 NaN dtype: float64
>>> ser.isna() 0 False 1 False 2 True dtype: bool
- isnull()[source]¶
Series.isnull is an alias for Series.isna.
Detect missing values.
Return a boolean same-sized object indicating if the values are NA. NA values, such as None or
numpy.NaN
, gets mapped to True values. Everything else gets mapped to False values. Characters such as empty strings''
ornumpy.inf
are not considered NA values (unless you setpandas.options.mode.use_inf_as_na = True
).- Returns
Mask of bool values for each element in Series that indicates whether an element is an NA value.
- Return type
See also
Series.isnull
Alias of isna.
Series.notna
Boolean inverse of isna.
Series.dropna
Omit axes labels with missing values.
isna
Top-level isna.
Examples
Show which entries in a DataFrame are NA.
>>> df = pd.DataFrame(dict(age=[5, 6, np.NaN], ... born=[pd.NaT, pd.Timestamp('1939-05-27'), ... pd.Timestamp('1940-04-25')], ... name=['Alfred', 'Batman', ''], ... toy=[None, 'Batmobile', 'Joker'])) >>> df age born name toy 0 5.0 NaT Alfred None 1 6.0 1939-05-27 Batman Batmobile 2 NaN 1940-04-25 Joker
>>> df.isna() age born name toy 0 False True False True 1 False False False False 2 True False False False
Show which entries in a Series are NA.
>>> ser = pd.Series([5, 6, np.NaN]) >>> ser 0 5.0 1 6.0 2 NaN dtype: float64
>>> ser.isna() 0 False 1 False 2 True dtype: bool
- items()[source]¶
Lazily iterate over (index, value) tuples.
This method returns an iterable tuple (index, value). This is convenient if you want to create a lazy iterator.
- Returns
Iterable of tuples containing the (index, value) pairs from a Series.
- Return type
iterable
See also
DataFrame.items
Iterate over (column name, Series) pairs.
DataFrame.iterrows
Iterate over DataFrame rows as (index, Series) pairs.
Examples
>>> s = pd.Series(['A', 'B', 'C']) >>> for index, value in s.items(): ... print(f"Index : {index}, Value : {value}") Index : 0, Value : A Index : 1, Value : B Index : 2, Value : C
- iteritems()[source]¶
Lazily iterate over (index, value) tuples.
This method returns an iterable tuple (index, value). This is convenient if you want to create a lazy iterator.
- Returns
Iterable of tuples containing the (index, value) pairs from a Series.
- Return type
iterable
See also
DataFrame.items
Iterate over (column name, Series) pairs.
DataFrame.iterrows
Iterate over DataFrame rows as (index, Series) pairs.
Examples
>>> s = pd.Series(['A', 'B', 'C']) >>> for index, value in s.items(): ... print(f"Index : {index}, Value : {value}") Index : 0, Value : A Index : 1, Value : B Index : 2, Value : C
- kurt(axis: Axis | None | lib.NoDefault = NoDefault.no_default, skipna=True, level=None, numeric_only=None, **kwargs)¶
Return unbiased kurtosis over requested axis.
Kurtosis obtained using Fisher’s definition of kurtosis (kurtosis of normal == 0.0). Normalized by N-1.
- Parameters
axis ({index (0)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
**kwargs – Additional keyword arguments to be passed to the function.
- Return type
scalar or Series (if level specified)
- kurtosis(axis: Axis | None | lib.NoDefault = NoDefault.no_default, skipna=True, level=None, numeric_only=None, **kwargs)¶
Return unbiased kurtosis over requested axis.
Kurtosis obtained using Fisher’s definition of kurtosis (kurtosis of normal == 0.0). Normalized by N-1.
- Parameters
axis ({index (0)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
**kwargs – Additional keyword arguments to be passed to the function.
- Return type
scalar or Series (if level specified)
- le(other, level=None, fill_value=None, axis=0)¶
Return Less than or equal to of series and other, element-wise (binary operator le).
Equivalent to
series <= other
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
Examples
>>> a = pd.Series([1, 1, 1, np.nan, 1], index=['a', 'b', 'c', 'd', 'e']) >>> a a 1.0 b 1.0 c 1.0 d NaN e 1.0 dtype: float64 >>> b = pd.Series([0, 1, 2, np.nan, 1], index=['a', 'b', 'c', 'd', 'f']) >>> b a 0.0 b 1.0 c 2.0 d NaN f 1.0 dtype: float64 >>> a.le(b, fill_value=0) a False b True c True d False e False f True dtype: bool
- lt(other, level=None, fill_value=None, axis=0)¶
Return Less than of series and other, element-wise (binary operator lt).
Equivalent to
series < other
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
Examples
>>> a = pd.Series([1, 1, 1, np.nan, 1], index=['a', 'b', 'c', 'd', 'e']) >>> a a 1.0 b 1.0 c 1.0 d NaN e 1.0 dtype: float64 >>> b = pd.Series([0, 1, 2, np.nan, 1], index=['a', 'b', 'c', 'd', 'f']) >>> b a 0.0 b 1.0 c 2.0 d NaN f 1.0 dtype: float64 >>> a.lt(b, fill_value=0) a False b False c True d False e False f True dtype: bool
- mad(axis=None, skipna=True, level=None)¶
Return the mean absolute deviation of the values over the requested axis.
- Parameters
axis ({index (0)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar.
- Return type
scalar or Series (if level specified)
- map(arg, na_action=None)[source]¶
Map values of Series according to an input mapping or function.
Used for substituting each value in a Series with another value, that may be derived from a function, a
dict
or aSeries
.- Parameters
arg (function, collections.abc.Mapping subclass or Series) – Mapping correspondence.
na_action ({None, ‘ignore’}, default None) – If ‘ignore’, propagate NaN values, without passing them to the mapping correspondence.
- Returns
Same index as caller.
- Return type
See also
Series.apply
For applying more complex functions on a Series.
DataFrame.apply
Apply a function row-/column-wise.
DataFrame.applymap
Apply a function elementwise on a whole DataFrame.
Notes
When
arg
is a dictionary, values in Series that are not in the dictionary (as keys) are converted toNaN
. However, if the dictionary is adict
subclass that defines__missing__
(i.e. provides a method for default values), then this default is used rather thanNaN
.Examples
>>> s = pd.Series(['cat', 'dog', np.nan, 'rabbit']) >>> s 0 cat 1 dog 2 NaN 3 rabbit dtype: object
map
accepts adict
or aSeries
. Values that are not found in thedict
are converted toNaN
, unless the dict has a default value (e.g.defaultdict
):>>> s.map({'cat': 'kitten', 'dog': 'puppy'}) 0 kitten 1 puppy 2 NaN 3 NaN dtype: object
It also accepts a function:
>>> s.map('I am a {}'.format) 0 I am a cat 1 I am a dog 2 I am a nan 3 I am a rabbit dtype: object
To avoid applying the function to missing values (and keep them as
NaN
)na_action='ignore'
can be used:>>> s.map('I am a {}'.format, na_action='ignore') 0 I am a cat 1 I am a dog 2 NaN 3 I am a rabbit dtype: object
- mask(cond, other=nan, inplace=False, axis=None, level=None, errors=NoDefault.no_default, try_cast=NoDefault.no_default)[source]¶
Replace values where the condition is True.
- Parameters
cond (bool Series/DataFrame, array-like, or callable) – Where cond is False, keep the original value. Where True, replace with corresponding value from other. If cond is callable, it is computed on the Series/DataFrame and should return boolean Series/DataFrame or array. The callable must not change input Series/DataFrame (though pandas doesn’t check it).
other (scalar, Series/DataFrame, or callable) – Entries where cond is True are replaced with corresponding value from other. If other is callable, it is computed on the Series/DataFrame and should return scalar or Series/DataFrame. The callable must not change input Series/DataFrame (though pandas doesn’t check it).
inplace (bool, default False) – Whether to perform the operation in place on the data.
axis (int, default None) – Alignment axis if needed.
level (int, default None) – Alignment level if needed.
errors (str, {‘raise’, ‘ignore’}, default ‘raise’) – Note that currently this parameter won’t affect the results and will always coerce to a suitable dtype.
‘raise’ : allow exceptions to be raised.
‘ignore’ : suppress exceptions. On error return original object.
try_cast (bool, default None) – Try to cast the result back to the input type (if possible).
Deprecated since version 1.3.0: Manually cast back if necessary.
- Return type
Same type as caller or None if
inplace=True
.
See also
DataFrame.where()
Return an object of same shape as self.
Notes
The mask method is an application of the if-then idiom. For each element in the calling DataFrame, if
cond
isFalse
the element is used; otherwise the corresponding element from the DataFrameother
is used.The signature for
DataFrame.where()
differs fromnumpy.where()
. Roughlydf1.where(m, df2)
is equivalent tonp.where(m, df1, df2)
.For further details and examples see the
mask
documentation in indexing.Examples
>>> s = pd.Series(range(5)) >>> s.where(s > 0) 0 NaN 1 1.0 2 2.0 3 3.0 4 4.0 dtype: float64 >>> s.mask(s > 0) 0 0.0 1 NaN 2 NaN 3 NaN 4 NaN dtype: float64
>>> s.where(s > 1, 10) 0 10 1 10 2 2 3 3 4 4 dtype: int64 >>> s.mask(s > 1, 10) 0 0 1 1 2 10 3 10 4 10 dtype: int64
>>> df = pd.DataFrame(np.arange(10).reshape(-1, 2), columns=['A', 'B']) >>> df A B 0 0 1 1 2 3 2 4 5 3 6 7 4 8 9 >>> m = df % 3 == 0 >>> df.where(m, -df) A B 0 0 -1 1 -2 3 2 -4 -5 3 6 -7 4 -8 9 >>> df.where(m, -df) == np.where(m, df, -df) A B 0 True True 1 True True 2 True True 3 True True 4 True True >>> df.where(m, -df) == df.mask(~m, -df) A B 0 True True 1 True True 2 True True 3 True True 4 True True
- max(axis: int | None | lib.NoDefault = NoDefault.no_default, skipna=True, level=None, numeric_only=None, **kwargs)¶
Return the maximum of the values over the requested axis.
If you want the index of the maximum, use
idxmax
. This is the equivalent of thenumpy.ndarray
methodargmax
.- Parameters
axis ({index (0)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
**kwargs – Additional keyword arguments to be passed to the function.
- Return type
scalar or Series (if level specified)
See also
Series.sum
Return the sum.
Series.min
Return the minimum.
Series.max
Return the maximum.
Series.idxmin
Return the index of the minimum.
Series.idxmax
Return the index of the maximum.
DataFrame.sum
Return the sum over the requested axis.
DataFrame.min
Return the minimum over the requested axis.
DataFrame.max
Return the maximum over the requested axis.
DataFrame.idxmin
Return the index of the minimum over the requested axis.
DataFrame.idxmax
Return the index of the maximum over the requested axis.
Examples
>>> idx = pd.MultiIndex.from_arrays([ ... ['warm', 'warm', 'cold', 'cold'], ... ['dog', 'falcon', 'fish', 'spider']], ... names=['blooded', 'animal']) >>> s = pd.Series([4, 2, 0, 8], name='legs', index=idx) >>> s blooded animal warm dog 4 falcon 2 cold fish 0 spider 8 Name: legs, dtype: int64
>>> s.max() 8
- mean(axis: int | None | lib.NoDefault = NoDefault.no_default, skipna=True, level=None, numeric_only=None, **kwargs)¶
Return the mean of the values over the requested axis.
- Parameters
axis ({index (0)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
**kwargs – Additional keyword arguments to be passed to the function.
- Return type
scalar or Series (if level specified)
- median(axis: int | None | lib.NoDefault = NoDefault.no_default, skipna=True, level=None, numeric_only=None, **kwargs)¶
Return the median of the values over the requested axis.
- Parameters
axis ({index (0)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
**kwargs – Additional keyword arguments to be passed to the function.
- Return type
scalar or Series (if level specified)
- memory_usage(index=True, deep=False)[source]¶
Return the memory usage of the Series.
The memory usage can optionally include the contribution of the index and of elements of object dtype.
- Parameters
index (bool, default True) – Specifies whether to include the memory usage of the Series index.
deep (bool, default False) – If True, introspect the data deeply by interrogating object dtypes for system-level memory consumption, and include it in the returned value.
- Returns
Bytes of memory consumed.
- Return type
See also
numpy.ndarray.nbytes
Total bytes consumed by the elements of the array.
DataFrame.memory_usage
Bytes consumed by a DataFrame.
Examples
>>> s = pd.Series(range(3)) >>> s.memory_usage() 152
Not including the index gives the size of the rest of the data, which is necessarily smaller:
>>> s.memory_usage(index=False) 24
The memory footprint of object values is ignored by default:
>>> s = pd.Series(["a", "b"]) >>> s.values array(['a', 'b'], dtype=object) >>> s.memory_usage() 144 >>> s.memory_usage(deep=True) 244
- min(axis: int | None | lib.NoDefault = NoDefault.no_default, skipna=True, level=None, numeric_only=None, **kwargs)¶
Return the minimum of the values over the requested axis.
If you want the index of the minimum, use
idxmin
. This is the equivalent of thenumpy.ndarray
methodargmin
.- Parameters
axis ({index (0)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
**kwargs – Additional keyword arguments to be passed to the function.
- Return type
scalar or Series (if level specified)
See also
Series.sum
Return the sum.
Series.min
Return the minimum.
Series.max
Return the maximum.
Series.idxmin
Return the index of the minimum.
Series.idxmax
Return the index of the maximum.
DataFrame.sum
Return the sum over the requested axis.
DataFrame.min
Return the minimum over the requested axis.
DataFrame.max
Return the maximum over the requested axis.
DataFrame.idxmin
Return the index of the minimum over the requested axis.
DataFrame.idxmax
Return the index of the maximum over the requested axis.
Examples
>>> idx = pd.MultiIndex.from_arrays([ ... ['warm', 'warm', 'cold', 'cold'], ... ['dog', 'falcon', 'fish', 'spider']], ... names=['blooded', 'animal']) >>> s = pd.Series([4, 2, 0, 8], name='legs', index=idx) >>> s blooded animal warm dog 4 falcon 2 cold fish 0 spider 8 Name: legs, dtype: int64
>>> s.min() 0
- mod(other, level=None, fill_value=None, axis=0)¶
Return Modulo of series and other, element-wise (binary operator mod).
Equivalent to
series % other
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.rmod
Reverse of the Modulo operator, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.mod(b, fill_value=0) a 0.0 b NaN c NaN d 0.0 e NaN dtype: float64
- mode(dropna=True)[source]¶
Return the mode(s) of the Series.
The mode is the value that appears most often. There can be multiple modes.
Always returns Series even if only one value is returned.
- Parameters
dropna (bool, default True) – Don’t consider counts of NaN/NaT.
- Returns
Modes of the Series in sorted order.
- Return type
- mul(other, level=None, fill_value=None, axis=0)¶
Return Multiplication of series and other, element-wise (binary operator mul).
Equivalent to
series * other
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.rmul
Reverse of the Multiplication operator, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.multiply(b, fill_value=0) a 1.0 b 0.0 c 0.0 d 0.0 e NaN dtype: float64
- multiply(other, level=None, fill_value=None, axis=0)¶
Return Multiplication of series and other, element-wise (binary operator mul).
Equivalent to
series * other
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.rmul
Reverse of the Multiplication operator, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.multiply(b, fill_value=0) a 1.0 b 0.0 c 0.0 d 0.0 e NaN dtype: float64
- property name: Hashable¶
Return the name of the Series.
The name of a Series becomes its index or column name if it is used to form a DataFrame. It is also used whenever displaying the Series using the interpreter.
- Returns
The name of the Series, also the column name if part of a DataFrame.
- Return type
label (hashable object)
See also
Series.rename
Sets the Series name when given a scalar input.
Index.name
Corresponding Index property.
Examples
The Series name can be set initially when calling the constructor.
>>> s = pd.Series([1, 2, 3], dtype=np.int64, name='Numbers') >>> s 0 1 1 2 2 3 Name: Numbers, dtype: int64 >>> s.name = "Integers" >>> s 0 1 1 2 2 3 Name: Integers, dtype: int64
The name of a Series within a DataFrame is its column name.
>>> df = pd.DataFrame([[1, 2], [3, 4], [5, 6]], ... columns=["Odd Numbers", "Even Numbers"]) >>> df Odd Numbers Even Numbers 0 1 2 1 3 4 2 5 6 >>> df["Even Numbers"].name 'Even Numbers'
- ne(other, level=None, fill_value=None, axis=0)¶
Return Not equal to of series and other, element-wise (binary operator ne).
Equivalent to
series != other
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.ne(b, fill_value=0) a False b True c True d True e True dtype: bool
- nlargest(n=5, keep='first')[source]¶
Return the largest n elements.
- Parameters
n (int, default 5) – Return this many descending sorted values.
keep ({‘first’, ‘last’, ‘all’}, default ‘first’) – When there are duplicate values that cannot all fit in a Series of n elements:
first
: return the first n occurrences in order of appearance.last
: return the last n occurrences in reverse order of appearance.all
: keep all occurrences. This can result in a Series of size larger than n.
- Returns
The n largest values in the Series, sorted in decreasing order.
- Return type
See also
Series.nsmallest
Get the n smallest elements.
Series.sort_values
Sort Series by values.
Series.head
Return the first n rows.
Notes
Faster than
.sort_values(ascending=False).head(n)
for small n relative to the size of theSeries
object.Examples
>>> countries_population = {"Italy": 59000000, "France": 65000000, ... "Malta": 434000, "Maldives": 434000, ... "Brunei": 434000, "Iceland": 337000, ... "Nauru": 11300, "Tuvalu": 11300, ... "Anguilla": 11300, "Montserrat": 5200} >>> s = pd.Series(countries_population) >>> s Italy 59000000 France 65000000 Malta 434000 Maldives 434000 Brunei 434000 Iceland 337000 Nauru 11300 Tuvalu 11300 Anguilla 11300 Montserrat 5200 dtype: int64
The n largest elements where
n=5
by default.>>> s.nlargest() France 65000000 Italy 59000000 Malta 434000 Maldives 434000 Brunei 434000 dtype: int64
The n largest elements where
n=3
. Default keep value is ‘first’ so Malta will be kept.>>> s.nlargest(3) France 65000000 Italy 59000000 Malta 434000 dtype: int64
The n largest elements where
n=3
and keeping the last duplicates. Brunei will be kept since it is the last with value 434000 based on the index order.>>> s.nlargest(3, keep='last') France 65000000 Italy 59000000 Brunei 434000 dtype: int64
The n largest elements where
n=3
with all duplicates kept. Note that the returned Series has five elements due to the three duplicates.>>> s.nlargest(3, keep='all') France 65000000 Italy 59000000 Malta 434000 Maldives 434000 Brunei 434000 dtype: int64
- notna()[source]¶
Detect existing (non-missing) values.
Return a boolean same-sized object indicating if the values are not NA. Non-missing values get mapped to True. Characters such as empty strings
''
ornumpy.inf
are not considered NA values (unless you setpandas.options.mode.use_inf_as_na = True
). NA values, such as None ornumpy.NaN
, get mapped to False values.- Returns
Mask of bool values for each element in Series that indicates whether an element is not an NA value.
- Return type
See also
Series.notnull
Alias of notna.
Series.isna
Boolean inverse of notna.
Series.dropna
Omit axes labels with missing values.
notna
Top-level notna.
Examples
Show which entries in a DataFrame are not NA.
>>> df = pd.DataFrame(dict(age=[5, 6, np.NaN], ... born=[pd.NaT, pd.Timestamp('1939-05-27'), ... pd.Timestamp('1940-04-25')], ... name=['Alfred', 'Batman', ''], ... toy=[None, 'Batmobile', 'Joker'])) >>> df age born name toy 0 5.0 NaT Alfred None 1 6.0 1939-05-27 Batman Batmobile 2 NaN 1940-04-25 Joker
>>> df.notna() age born name toy 0 True False True False 1 True True True True 2 False True True True
Show which entries in a Series are not NA.
>>> ser = pd.Series([5, 6, np.NaN]) >>> ser 0 5.0 1 6.0 2 NaN dtype: float64
>>> ser.notna() 0 True 1 True 2 False dtype: bool
- notnull()[source]¶
Series.notnull is an alias for Series.notna.
Detect existing (non-missing) values.
Return a boolean same-sized object indicating if the values are not NA. Non-missing values get mapped to True. Characters such as empty strings
''
ornumpy.inf
are not considered NA values (unless you setpandas.options.mode.use_inf_as_na = True
). NA values, such as None ornumpy.NaN
, get mapped to False values.- Returns
Mask of bool values for each element in Series that indicates whether an element is not an NA value.
- Return type
See also
Series.notnull
Alias of notna.
Series.isna
Boolean inverse of notna.
Series.dropna
Omit axes labels with missing values.
notna
Top-level notna.
Examples
Show which entries in a DataFrame are not NA.
>>> df = pd.DataFrame(dict(age=[5, 6, np.NaN], ... born=[pd.NaT, pd.Timestamp('1939-05-27'), ... pd.Timestamp('1940-04-25')], ... name=['Alfred', 'Batman', ''], ... toy=[None, 'Batmobile', 'Joker'])) >>> df age born name toy 0 5.0 NaT Alfred None 1 6.0 1939-05-27 Batman Batmobile 2 NaN 1940-04-25 Joker
>>> df.notna() age born name toy 0 True False True False 1 True True True True 2 False True True True
Show which entries in a Series are not NA.
>>> ser = pd.Series([5, 6, np.NaN]) >>> ser 0 5.0 1 6.0 2 NaN dtype: float64
>>> ser.notna() 0 True 1 True 2 False dtype: bool
- nsmallest(n=5, keep='first')[source]¶
Return the smallest n elements.
- Parameters
n (int, default 5) – Return this many ascending sorted values.
keep ({‘first’, ‘last’, ‘all’}, default ‘first’) – When there are duplicate values that cannot all fit in a Series of n elements:
first
: return the first n occurrences in order of appearance.last
: return the last n occurrences in reverse order of appearance.all
: keep all occurrences. This can result in a Series of size larger than n.
- Returns
The n smallest values in the Series, sorted in increasing order.
- Return type
See also
Series.nlargest
Get the n largest elements.
Series.sort_values
Sort Series by values.
Series.head
Return the first n rows.
Notes
Faster than
.sort_values().head(n)
for small n relative to the size of theSeries
object.Examples
>>> countries_population = {"Italy": 59000000, "France": 65000000, ... "Brunei": 434000, "Malta": 434000, ... "Maldives": 434000, "Iceland": 337000, ... "Nauru": 11300, "Tuvalu": 11300, ... "Anguilla": 11300, "Montserrat": 5200} >>> s = pd.Series(countries_population) >>> s Italy 59000000 France 65000000 Brunei 434000 Malta 434000 Maldives 434000 Iceland 337000 Nauru 11300 Tuvalu 11300 Anguilla 11300 Montserrat 5200 dtype: int64
The n smallest elements where
n=5
by default.>>> s.nsmallest() Montserrat 5200 Nauru 11300 Tuvalu 11300 Anguilla 11300 Iceland 337000 dtype: int64
The n smallest elements where
n=3
. Default keep value is ‘first’ so Nauru and Tuvalu will be kept.>>> s.nsmallest(3) Montserrat 5200 Nauru 11300 Tuvalu 11300 dtype: int64
The n smallest elements where
n=3
and keeping the last duplicates. Anguilla and Tuvalu will be kept since they are the last with value 11300 based on the index order.>>> s.nsmallest(3, keep='last') Montserrat 5200 Anguilla 11300 Tuvalu 11300 dtype: int64
The n smallest elements where
n=3
with all duplicates kept. Note that the returned Series has four elements due to the three duplicates.>>> s.nsmallest(3, keep='all') Montserrat 5200 Nauru 11300 Tuvalu 11300 Anguilla 11300 dtype: int64
- plot¶
alias of
pandas.plotting._core.PlotAccessor
- pop(item)[source]¶
Return item and drops from series. Raise KeyError if not found.
- Parameters
item (label) – Index of the element that needs to be removed.
- Return type
Value that is popped from series.
Examples
>>> ser = pd.Series([1,2,3])
>>> ser.pop(0) 1
>>> ser 1 2 2 3 dtype: int64
- pow(other, level=None, fill_value=None, axis=0)¶
Return Exponential power of series and other, element-wise (binary operator pow).
Equivalent to
series ** other
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.rpow
Reverse of the Exponential power operator, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.pow(b, fill_value=0) a 1.0 b 1.0 c 1.0 d 0.0 e NaN dtype: float64
- prod(axis=None, skipna=True, level=None, numeric_only=None, min_count=0, **kwargs)¶
Return the product of the values over the requested axis.
- Parameters
axis ({index (0)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
min_count (int, default 0) – The required number of valid values to perform the operation. If fewer than
min_count
non-NA values are present the result will be NA.**kwargs – Additional keyword arguments to be passed to the function.
- Return type
scalar or Series (if level specified)
See also
Series.sum
Return the sum.
Series.min
Return the minimum.
Series.max
Return the maximum.
Series.idxmin
Return the index of the minimum.
Series.idxmax
Return the index of the maximum.
DataFrame.sum
Return the sum over the requested axis.
DataFrame.min
Return the minimum over the requested axis.
DataFrame.max
Return the maximum over the requested axis.
DataFrame.idxmin
Return the index of the minimum over the requested axis.
DataFrame.idxmax
Return the index of the maximum over the requested axis.
Examples
By default, the product of an empty or all-NA Series is
1
>>> pd.Series([], dtype="float64").prod() 1.0
This can be controlled with the
min_count
parameter>>> pd.Series([], dtype="float64").prod(min_count=1) nan
Thanks to the
skipna
parameter,min_count
handles all-NA and empty series identically.>>> pd.Series([np.nan]).prod() 1.0
>>> pd.Series([np.nan]).prod(min_count=1) nan
- product(axis=None, skipna=True, level=None, numeric_only=None, min_count=0, **kwargs)¶
Return the product of the values over the requested axis.
- Parameters
axis ({index (0)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
min_count (int, default 0) – The required number of valid values to perform the operation. If fewer than
min_count
non-NA values are present the result will be NA.**kwargs – Additional keyword arguments to be passed to the function.
- Return type
scalar or Series (if level specified)
See also
Series.sum
Return the sum.
Series.min
Return the minimum.
Series.max
Return the maximum.
Series.idxmin
Return the index of the minimum.
Series.idxmax
Return the index of the maximum.
DataFrame.sum
Return the sum over the requested axis.
DataFrame.min
Return the minimum over the requested axis.
DataFrame.max
Return the maximum over the requested axis.
DataFrame.idxmin
Return the index of the minimum over the requested axis.
DataFrame.idxmax
Return the index of the maximum over the requested axis.
Examples
By default, the product of an empty or all-NA Series is
1
>>> pd.Series([], dtype="float64").prod() 1.0
This can be controlled with the
min_count
parameter>>> pd.Series([], dtype="float64").prod(min_count=1) nan
Thanks to the
skipna
parameter,min_count
handles all-NA and empty series identically.>>> pd.Series([np.nan]).prod() 1.0
>>> pd.Series([np.nan]).prod(min_count=1) nan
- quantile(q=0.5, interpolation='linear')[source]¶
Return value at the given quantile.
- Parameters
q (float or array-like, default 0.5 (50% quantile)) – The quantile(s) to compute, which can lie in range: 0 <= q <= 1.
interpolation ({‘linear’, ‘lower’, ‘higher’, ‘midpoint’, ‘nearest’}) – This optional parameter specifies the interpolation method to use, when the desired quantile lies between two data points i and j:
linear: i + (j - i) * fraction, where fraction is the fractional part of the index surrounded by i and j.
lower: i.
higher: j.
nearest: i or j whichever is nearest.
midpoint: (i + j) / 2.
- Returns
If
q
is an array, a Series will be returned where the index isq
and the values are the quantiles, otherwise a float will be returned.- Return type
See also
core.window.Rolling.quantile
Calculate the rolling quantile.
numpy.percentile
Returns the q-th percentile(s) of the array elements.
Examples
>>> s = pd.Series([1, 2, 3, 4]) >>> s.quantile(.5) 2.5 >>> s.quantile([.25, .5, .75]) 0.25 1.75 0.50 2.50 0.75 3.25 dtype: float64
- radd(other, level=None, fill_value=None, axis=0)¶
Return Addition of series and other, element-wise (binary operator radd).
Equivalent to
other + series
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.add
Element-wise Addition, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.add(b, fill_value=0) a 2.0 b 1.0 c 1.0 d 1.0 e NaN dtype: float64
- ravel(order='C')[source]¶
Return the flattened underlying data as an ndarray.
- Returns
Flattened data of the Series.
- Return type
numpy.ndarray or ndarray-like
See also
numpy.ndarray.ravel
Return a flattened array.
- rdiv(other, level=None, fill_value=None, axis=0)¶
Return Floating division of series and other, element-wise (binary operator rtruediv).
Equivalent to
other / series
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.truediv
Element-wise Floating division, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.divide(b, fill_value=0) a 1.0 b inf c inf d 0.0 e NaN dtype: float64
- rdivmod(other, level=None, fill_value=None, axis=0)¶
Return Integer division and modulo of series and other, element-wise (binary operator rdivmod).
Equivalent to
other divmod series
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
2-Tuple of Series
See also
Series.divmod
Element-wise Integer division and modulo, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.divmod(b, fill_value=0) (a 1.0 b NaN c NaN d 0.0 e NaN dtype: float64, a 0.0 b NaN c NaN d 0.0 e NaN dtype: float64)
- reindex(*args, **kwargs)[source]¶
Conform Series to new index with optional filling logic.
Places NA/NaN in locations having no value in the previous index. A new object is produced unless the new index is equivalent to the current one and
copy=False
.- Parameters
index (array-like, optional) – New labels / index to conform to, should be specified using keywords. Preferably an Index object to avoid duplicating data.
method ({None, ‘backfill’/’bfill’, ‘pad’/’ffill’, ‘nearest’}) – Method to use for filling holes in reindexed DataFrame. Please note: this is only applicable to DataFrames/Series with a monotonically increasing/decreasing index.
None (default): don’t fill gaps
pad / ffill: Propagate last valid observation forward to next valid.
backfill / bfill: Use next valid observation to fill gap.
nearest: Use nearest valid observations to fill gap.
copy (bool, default True) – Return a new object, even if the passed indexes are the same.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
fill_value (scalar, default np.NaN) – Value to use for missing values. Defaults to NaN, but can be any “compatible” value.
limit (int, default None) – Maximum number of consecutive elements to forward or backward fill.
tolerance (optional) – Maximum distance between original and new labels for inexact matches. The values of the index at the matching locations most satisfy the equation
abs(index[indexer] - target) <= tolerance
.Tolerance may be a scalar value, which applies the same tolerance to all values, or list-like, which applies variable tolerance per element. List-like includes list, tuple, array, Series, and must be the same size as the index and its dtype must exactly match the index’s type.
- Return type
Series with changed index.
See also
DataFrame.set_index
Set row labels.
DataFrame.reset_index
Remove row labels or move them to new columns.
DataFrame.reindex_like
Change to same indices as other DataFrame.
Examples
DataFrame.reindex
supports two calling conventions(index=index_labels, columns=column_labels, ...)
(labels, axis={'index', 'columns'}, ...)
We highly recommend using keyword arguments to clarify your intent.
Create a dataframe with some fictional data.
>>> index = ['Firefox', 'Chrome', 'Safari', 'IE10', 'Konqueror'] >>> df = pd.DataFrame({'http_status': [200, 200, 404, 404, 301], ... 'response_time': [0.04, 0.02, 0.07, 0.08, 1.0]}, ... index=index) >>> df http_status response_time Firefox 200 0.04 Chrome 200 0.02 Safari 404 0.07 IE10 404 0.08 Konqueror 301 1.00
Create a new index and reindex the dataframe. By default values in the new index that do not have corresponding records in the dataframe are assigned
NaN
.>>> new_index = ['Safari', 'Iceweasel', 'Comodo Dragon', 'IE10', ... 'Chrome'] >>> df.reindex(new_index) http_status response_time Safari 404.0 0.07 Iceweasel NaN NaN Comodo Dragon NaN NaN IE10 404.0 0.08 Chrome 200.0 0.02
We can fill in the missing values by passing a value to the keyword
fill_value
. Because the index is not monotonically increasing or decreasing, we cannot use arguments to the keywordmethod
to fill theNaN
values.>>> df.reindex(new_index, fill_value=0) http_status response_time Safari 404 0.07 Iceweasel 0 0.00 Comodo Dragon 0 0.00 IE10 404 0.08 Chrome 200 0.02
>>> df.reindex(new_index, fill_value='missing') http_status response_time Safari 404 0.07 Iceweasel missing missing Comodo Dragon missing missing IE10 404 0.08 Chrome 200 0.02
We can also reindex the columns.
>>> df.reindex(columns=['http_status', 'user_agent']) http_status user_agent Firefox 200 NaN Chrome 200 NaN Safari 404 NaN IE10 404 NaN Konqueror 301 NaN
Or we can use “axis-style” keyword arguments
>>> df.reindex(['http_status', 'user_agent'], axis="columns") http_status user_agent Firefox 200 NaN Chrome 200 NaN Safari 404 NaN IE10 404 NaN Konqueror 301 NaN
To further illustrate the filling functionality in
reindex
, we will create a dataframe with a monotonically increasing index (for example, a sequence of dates).>>> date_index = pd.date_range('1/1/2010', periods=6, freq='D') >>> df2 = pd.DataFrame({"prices": [100, 101, np.nan, 100, 89, 88]}, ... index=date_index) >>> df2 prices 2010-01-01 100.0 2010-01-02 101.0 2010-01-03 NaN 2010-01-04 100.0 2010-01-05 89.0 2010-01-06 88.0
Suppose we decide to expand the dataframe to cover a wider date range.
>>> date_index2 = pd.date_range('12/29/2009', periods=10, freq='D') >>> df2.reindex(date_index2) prices 2009-12-29 NaN 2009-12-30 NaN 2009-12-31 NaN 2010-01-01 100.0 2010-01-02 101.0 2010-01-03 NaN 2010-01-04 100.0 2010-01-05 89.0 2010-01-06 88.0 2010-01-07 NaN
The index entries that did not have a value in the original data frame (for example, ‘2009-12-29’) are by default filled with
NaN
. If desired, we can fill in the missing values using one of several options.For example, to back-propagate the last valid value to fill the
NaN
values, passbfill
as an argument to themethod
keyword.>>> df2.reindex(date_index2, method='bfill') prices 2009-12-29 100.0 2009-12-30 100.0 2009-12-31 100.0 2010-01-01 100.0 2010-01-02 101.0 2010-01-03 NaN 2010-01-04 100.0 2010-01-05 89.0 2010-01-06 88.0 2010-01-07 NaN
Please note that the
NaN
value present in the original dataframe (at index value 2010-01-03) will not be filled by any of the value propagation schemes. This is because filling while reindexing does not look at dataframe values, but only compares the original and desired indexes. If you do want to fill in theNaN
values present in the original dataframe, use thefillna()
method.See the user guide for more.
- rename(index=None, *, axis=None, copy=True, inplace=False, level=None, errors='ignore')[source]¶
Alter Series index labels or name.
Function / dict values must be unique (1-to-1). Labels not contained in a dict / Series will be left as-is. Extra labels listed don’t throw an error.
Alternatively, change
Series.name
with a scalar value.See the user guide for more.
- Parameters
axis ({0 or “index”}) – Unused. Accepted for compatibility with DataFrame method only.
index (scalar, hashable sequence, dict-like or function, optional) – Functions or dict-like are transformations to apply to the index. Scalar or hashable sequence-like will alter the
Series.name
attribute.**kwargs – Additional keyword arguments passed to the function. Only the “inplace” keyword is used.
- Returns
Series with index labels or name altered or None if
inplace=True
.- Return type
Series or None
See also
DataFrame.rename
Corresponding DataFrame method.
Series.rename_axis
Set the name of the axis.
Examples
>>> s = pd.Series([1, 2, 3]) >>> s 0 1 1 2 2 3 dtype: int64 >>> s.rename("my_name") # scalar, changes Series.name 0 1 1 2 2 3 Name: my_name, dtype: int64 >>> s.rename(lambda x: x ** 2) # function, changes labels 0 1 1 2 4 3 dtype: int64 >>> s.rename({1: 3, 2: 5}) # mapping, changes labels 0 1 3 2 5 3 dtype: int64
- reorder_levels(order)[source]¶
Rearrange index levels using input order.
May not drop or duplicate levels.
- Parameters
order (list of int representing new level order) – Reference level by number or key.
- Return type
type of caller (new object)
- repeat(repeats, axis=None)[source]¶
Repeat elements of a Series.
Returns a new Series where each element of the current Series is repeated consecutively a given number of times.
- Parameters
repeats (int or array of ints) – The number of repetitions for each element. This should be a non-negative integer. Repeating 0 times will return an empty Series.
axis (None) – Must be
None
. Has no effect but is accepted for compatibility with numpy.
- Returns
Newly created Series with repeated elements.
- Return type
See also
Index.repeat
Equivalent function for Index.
numpy.repeat
Similar method for
numpy.ndarray
.
Examples
>>> s = pd.Series(['a', 'b', 'c']) >>> s 0 a 1 b 2 c dtype: object >>> s.repeat(2) 0 a 0 a 1 b 1 b 2 c 2 c dtype: object >>> s.repeat([1, 2, 3]) 0 a 1 b 1 b 2 c 2 c 2 c dtype: object
- replace(to_replace=None, value=NoDefault.no_default, inplace=False, limit=None, regex=False, method=NoDefault.no_default)[source]¶
Replace values given in to_replace with value.
Values of the Series are replaced with other values dynamically.
This differs from updating with
.loc
or.iloc
, which require you to specify a location to update with some value.- Parameters
to_replace (str, regex, list, dict, Series, int, float, or None) – How to find the values that will be replaced.
numeric, str or regex:
numeric: numeric values equal to to_replace will be replaced with value
str: string exactly matching to_replace will be replaced with value
regex: regexs matching to_replace will be replaced with value
list of str, regex, or numeric:
First, if to_replace and value are both lists, they must be the same length.
Second, if
regex=True
then all of the strings in both lists will be interpreted as regexs otherwise they will match directly. This doesn’t matter much for value since there are only a few possible substitution regexes you can use.str, regex and numeric rules apply as above.
dict:
Dicts can be used to specify different replacement values for different existing values. For example,
{'a': 'b', 'y': 'z'}
replaces the value ‘a’ with ‘b’ and ‘y’ with ‘z’. To use a dict in this way the value parameter should be None.For a DataFrame a dict can specify that different values should be replaced in different columns. For example,
{'a': 1, 'b': 'z'}
looks for the value 1 in column ‘a’ and the value ‘z’ in column ‘b’ and replaces these values with whatever is specified in value. The value parameter should not beNone
in this case. You can treat this as a special case of passing two lists except that you are specifying the column to search in.For a DataFrame nested dictionaries, e.g.,
{'a': {'b': np.nan}}
, are read as follows: look in column ‘a’ for the value ‘b’ and replace it with NaN. The value parameter should beNone
to use a nested dict in this way. You can nest regular expressions as well. Note that column names (the top-level dictionary keys in a nested dictionary) cannot be regular expressions.
None:
This means that the regex argument must be a string, compiled regular expression, or list, dict, ndarray or Series of such elements. If value is also
None
then this must be a nested dictionary or Series.
See the examples section for examples of each of these.
value (scalar, dict, list, str, regex, default None) – Value to replace any values matching to_replace with. For a DataFrame a dict of values can be used to specify which value to use for each column (columns not in the dict will not be filled). Regular expressions, strings and lists or dicts of such objects are also allowed.
inplace (bool, default False) – If True, performs operation inplace and returns None.
limit (int, default None) – Maximum size gap to forward or backward fill.
regex (bool or same types as to_replace, default False) – Whether to interpret to_replace and/or value as regular expressions. If this is
True
then to_replace must be a string. Alternatively, this could be a regular expression or a list, dict, or array of regular expressions in which case to_replace must beNone
.method ({‘pad’, ‘ffill’, ‘bfill’, None}) – The method to use when for replacement, when to_replace is a scalar, list or tuple and value is
None
.Changed in version 0.23.0: Added to DataFrame.
- Returns
Object after replacement.
- Return type
- Raises
If regex is not a
bool
and to_replace is notNone
.
If to_replace is not a scalar, array-like,
dict
, orNone
* If to_replace is adict
and value is not alist
,dict
,ndarray
, orSeries
* If to_replace isNone
and regex is not compilable into a regular expression or is a list, dict, ndarray, or Series. * When replacing multiplebool
ordatetime64
objects and the arguments to to_replace does not match the type of the value being replaced
If a
list
or anndarray
is passed to to_replace and value but they are not the same length.
See also
Series.fillna
Fill NA values.
Series.where
Replace values based on boolean condition.
Series.str.replace
Simple string replacement.
Notes
Regex substitution is performed under the hood with
re.sub
. The rules for substitution forre.sub
are the same.Regular expressions will only substitute on strings, meaning you cannot provide, for example, a regular expression matching floating point numbers and expect the columns in your frame that have a numeric dtype to be matched. However, if those floating point numbers are strings, then you can do this.
This method has a lot of options. You are encouraged to experiment and play with this method to gain intuition about how it works.
When dict is used as the to_replace value, it is like key(s) in the dict are the to_replace part and value(s) in the dict are the value parameter.
Examples
Scalar `to_replace` and `value`
>>> s = pd.Series([1, 2, 3, 4, 5]) >>> s.replace(1, 5) 0 5 1 2 2 3 3 4 4 5 dtype: int64
>>> df = pd.DataFrame({'A': [0, 1, 2, 3, 4], ... 'B': [5, 6, 7, 8, 9], ... 'C': ['a', 'b', 'c', 'd', 'e']}) >>> df.replace(0, 5) A B C 0 5 5 a 1 1 6 b 2 2 7 c 3 3 8 d 4 4 9 e
List-like `to_replace`
>>> df.replace([0, 1, 2, 3], 4) A B C 0 4 5 a 1 4 6 b 2 4 7 c 3 4 8 d 4 4 9 e
>>> df.replace([0, 1, 2, 3], [4, 3, 2, 1]) A B C 0 4 5 a 1 3 6 b 2 2 7 c 3 1 8 d 4 4 9 e
>>> s.replace([1, 2], method='bfill') 0 3 1 3 2 3 3 4 4 5 dtype: int64
dict-like `to_replace`
>>> df.replace({0: 10, 1: 100}) A B C 0 10 5 a 1 100 6 b 2 2 7 c 3 3 8 d 4 4 9 e
>>> df.replace({'A': 0, 'B': 5}, 100) A B C 0 100 100 a 1 1 6 b 2 2 7 c 3 3 8 d 4 4 9 e
>>> df.replace({'A': {0: 100, 4: 400}}) A B C 0 100 5 a 1 1 6 b 2 2 7 c 3 3 8 d 4 400 9 e
Regular expression `to_replace`
>>> df = pd.DataFrame({'A': ['bat', 'foo', 'bait'], ... 'B': ['abc', 'bar', 'xyz']}) >>> df.replace(to_replace=r'^ba.$', value='new', regex=True) A B 0 new abc 1 foo new 2 bait xyz
>>> df.replace({'A': r'^ba.$'}, {'A': 'new'}, regex=True) A B 0 new abc 1 foo bar 2 bait xyz
>>> df.replace(regex=r'^ba.$', value='new') A B 0 new abc 1 foo new 2 bait xyz
>>> df.replace(regex={r'^ba.$': 'new', 'foo': 'xyz'}) A B 0 new abc 1 xyz new 2 bait xyz
>>> df.replace(regex=[r'^ba.$', 'foo'], value='new') A B 0 new abc 1 new new 2 bait xyz
Compare the behavior of
s.replace({'a': None})
ands.replace('a', None)
to understand the peculiarities of the to_replace parameter:>>> s = pd.Series([10, 'a', 'a', 'b', 'a'])
When one uses a dict as the to_replace value, it is like the value(s) in the dict are equal to the value parameter.
s.replace({'a': None})
is equivalent tos.replace(to_replace={'a': None}, value=None, method=None)
:>>> s.replace({'a': None}) 0 10 1 None 2 None 3 b 4 None dtype: object
When
value
is not explicitly passed and to_replace is a scalar, list or tuple, replace uses the method parameter (default ‘pad’) to do the replacement. So this is why the ‘a’ values are being replaced by 10 in rows 1 and 2 and ‘b’ in row 4 in this case.>>> s.replace('a') 0 10 1 10 2 10 3 b 4 b dtype: object
On the other hand, if
None
is explicitly passed forvalue
, it will be respected:>>> s.replace('a', None) 0 10 1 None 2 None 3 b 4 None dtype: object
Changed in version 1.4.0: Previously the explicit
None
was silently ignored.
- resample(rule, axis=0, closed=None, label=None, convention='start', kind=None, loffset=None, base=None, on=None, level=None, origin='start_day', offset=None)[source]¶
Resample time-series data.
Convenience method for frequency conversion and resampling of time series. The object must have a datetime-like index (DatetimeIndex, PeriodIndex, or TimedeltaIndex), or the caller must pass the label of a datetime-like series/index to the
on
/level
keyword parameter.- Parameters
rule (DateOffset, Timedelta or str) – The offset string or object representing target conversion.
axis ({0 or ‘index’, 1 or ‘columns’}, default 0) – Which axis to use for up- or down-sampling. For Series this will default to 0, i.e. along the rows. Must be DatetimeIndex, TimedeltaIndex or PeriodIndex.
closed ({‘right’, ‘left’}, default None) – Which side of bin interval is closed. The default is ‘left’ for all frequency offsets except for ‘M’, ‘A’, ‘Q’, ‘BM’, ‘BA’, ‘BQ’, and ‘W’ which all have a default of ‘right’.
label ({‘right’, ‘left’}, default None) – Which bin edge label to label bucket with. The default is ‘left’ for all frequency offsets except for ‘M’, ‘A’, ‘Q’, ‘BM’, ‘BA’, ‘BQ’, and ‘W’ which all have a default of ‘right’.
convention ({‘start’, ‘end’, ‘s’, ‘e’}, default ‘start’) – For PeriodIndex only, controls whether to use the start or end of rule.
kind ({‘timestamp’, ‘period’}, optional, default None) – Pass ‘timestamp’ to convert the resulting index to a DateTimeIndex or ‘period’ to convert it to a PeriodIndex. By default the input representation is retained.
loffset (timedelta, default None) – Adjust the resampled time labels.
Deprecated since version 1.1.0: You should add the loffset to the df.index after the resample. See below.
base (int, default 0) – For frequencies that evenly subdivide 1 day, the “origin” of the aggregated intervals. For example, for ‘5min’ frequency, base could range from 0 through 4. Defaults to 0.
Deprecated since version 1.1.0: The new arguments that you should use are ‘offset’ or ‘origin’.
on (str, optional) – For a DataFrame, column to use instead of index for resampling. Column must be datetime-like.
level (str or int, optional) – For a MultiIndex, level (name or number) to use for resampling. level must be datetime-like.
origin (Timestamp or str, default ‘start_day’) – The timestamp on which to adjust the grouping. The timezone of origin must match the timezone of the index. If string, must be one of the following:
‘epoch’: origin is 1970-01-01
‘start’: origin is the first value of the timeseries
‘start_day’: origin is the first day at midnight of the timeseries
New in version 1.1.0.
‘end’: origin is the last value of the timeseries
‘end_day’: origin is the ceiling midnight of the last day
New in version 1.3.0.
offset (Timedelta or str, default is None) – An offset timedelta added to the origin.
New in version 1.1.0.
- Returns
Resampler
object.- Return type
pandas.core.Resampler
See also
Series.resample
Resample a Series.
DataFrame.resample
Resample a DataFrame.
groupby
Group Series by mapping, function, label, or list of labels.
asfreq
Reindex a Series with the given frequency without grouping.
Notes
See the user guide for more.
To learn more about the offset strings, please see this link.
Examples
Start by creating a series with 9 one minute timestamps.
>>> index = pd.date_range('1/1/2000', periods=9, freq='T') >>> series = pd.Series(range(9), index=index) >>> series 2000-01-01 00:00:00 0 2000-01-01 00:01:00 1 2000-01-01 00:02:00 2 2000-01-01 00:03:00 3 2000-01-01 00:04:00 4 2000-01-01 00:05:00 5 2000-01-01 00:06:00 6 2000-01-01 00:07:00 7 2000-01-01 00:08:00 8 Freq: T, dtype: int64
Downsample the series into 3 minute bins and sum the values of the timestamps falling into a bin.
>>> series.resample('3T').sum() 2000-01-01 00:00:00 3 2000-01-01 00:03:00 12 2000-01-01 00:06:00 21 Freq: 3T, dtype: int64
Downsample the series into 3 minute bins as above, but label each bin using the right edge instead of the left. Please note that the value in the bucket used as the label is not included in the bucket, which it labels. For example, in the original series the bucket
2000-01-01 00:03:00
contains the value 3, but the summed value in the resampled bucket with the label2000-01-01 00:03:00
does not include 3 (if it did, the summed value would be 6, not 3). To include this value close the right side of the bin interval as illustrated in the example below this one.>>> series.resample('3T', label='right').sum() 2000-01-01 00:03:00 3 2000-01-01 00:06:00 12 2000-01-01 00:09:00 21 Freq: 3T, dtype: int64
Downsample the series into 3 minute bins as above, but close the right side of the bin interval.
>>> series.resample('3T', label='right', closed='right').sum() 2000-01-01 00:00:00 0 2000-01-01 00:03:00 6 2000-01-01 00:06:00 15 2000-01-01 00:09:00 15 Freq: 3T, dtype: int64
Upsample the series into 30 second bins.
>>> series.resample('30S').asfreq()[0:5] # Select first 5 rows 2000-01-01 00:00:00 0.0 2000-01-01 00:00:30 NaN 2000-01-01 00:01:00 1.0 2000-01-01 00:01:30 NaN 2000-01-01 00:02:00 2.0 Freq: 30S, dtype: float64
Upsample the series into 30 second bins and fill the
NaN
values using thepad
method.>>> series.resample('30S').pad()[0:5] 2000-01-01 00:00:00 0 2000-01-01 00:00:30 0 2000-01-01 00:01:00 1 2000-01-01 00:01:30 1 2000-01-01 00:02:00 2 Freq: 30S, dtype: int64
Upsample the series into 30 second bins and fill the
NaN
values using thebfill
method.>>> series.resample('30S').bfill()[0:5] 2000-01-01 00:00:00 0 2000-01-01 00:00:30 1 2000-01-01 00:01:00 1 2000-01-01 00:01:30 2 2000-01-01 00:02:00 2 Freq: 30S, dtype: int64
Pass a custom function via
apply
>>> def custom_resampler(arraylike): ... return np.sum(arraylike) + 5 ... >>> series.resample('3T').apply(custom_resampler) 2000-01-01 00:00:00 8 2000-01-01 00:03:00 17 2000-01-01 00:06:00 26 Freq: 3T, dtype: int64
For a Series with a PeriodIndex, the keyword convention can be used to control whether to use the start or end of rule.
Resample a year by quarter using ‘start’ convention. Values are assigned to the first quarter of the period.
>>> s = pd.Series([1, 2], index=pd.period_range('2012-01-01', ... freq='A', ... periods=2)) >>> s 2012 1 2013 2 Freq: A-DEC, dtype: int64 >>> s.resample('Q', convention='start').asfreq() 2012Q1 1.0 2012Q2 NaN 2012Q3 NaN 2012Q4 NaN 2013Q1 2.0 2013Q2 NaN 2013Q3 NaN 2013Q4 NaN Freq: Q-DEC, dtype: float64
Resample quarters by month using ‘end’ convention. Values are assigned to the last month of the period.
>>> q = pd.Series([1, 2, 3, 4], index=pd.period_range('2018-01-01', ... freq='Q', ... periods=4)) >>> q 2018Q1 1 2018Q2 2 2018Q3 3 2018Q4 4 Freq: Q-DEC, dtype: int64 >>> q.resample('M', convention='end').asfreq() 2018-03 1.0 2018-04 NaN 2018-05 NaN 2018-06 2.0 2018-07 NaN 2018-08 NaN 2018-09 3.0 2018-10 NaN 2018-11 NaN 2018-12 4.0 Freq: M, dtype: float64
For DataFrame objects, the keyword on can be used to specify the column instead of the index for resampling.
>>> d = {'price': [10, 11, 9, 13, 14, 18, 17, 19], ... 'volume': [50, 60, 40, 100, 50, 100, 40, 50]} >>> df = pd.DataFrame(d) >>> df['week_starting'] = pd.date_range('01/01/2018', ... periods=8, ... freq='W') >>> df price volume week_starting 0 10 50 2018-01-07 1 11 60 2018-01-14 2 9 40 2018-01-21 3 13 100 2018-01-28 4 14 50 2018-02-04 5 18 100 2018-02-11 6 17 40 2018-02-18 7 19 50 2018-02-25 >>> df.resample('M', on='week_starting').mean() price volume week_starting 2018-01-31 10.75 62.5 2018-02-28 17.00 60.0
For a DataFrame with MultiIndex, the keyword level can be used to specify on which level the resampling needs to take place.
>>> days = pd.date_range('1/1/2000', periods=4, freq='D') >>> d2 = {'price': [10, 11, 9, 13, 14, 18, 17, 19], ... 'volume': [50, 60, 40, 100, 50, 100, 40, 50]} >>> df2 = pd.DataFrame( ... d2, ... index=pd.MultiIndex.from_product( ... [days, ['morning', 'afternoon']] ... ) ... ) >>> df2 price volume 2000-01-01 morning 10 50 afternoon 11 60 2000-01-02 morning 9 40 afternoon 13 100 2000-01-03 morning 14 50 afternoon 18 100 2000-01-04 morning 17 40 afternoon 19 50 >>> df2.resample('D', level=0).sum() price volume 2000-01-01 21 110 2000-01-02 22 140 2000-01-03 32 150 2000-01-04 36 90
If you want to adjust the start of the bins based on a fixed timestamp:
>>> start, end = '2000-10-01 23:30:00', '2000-10-02 00:30:00' >>> rng = pd.date_range(start, end, freq='7min') >>> ts = pd.Series(np.arange(len(rng)) * 3, index=rng) >>> ts 2000-10-01 23:30:00 0 2000-10-01 23:37:00 3 2000-10-01 23:44:00 6 2000-10-01 23:51:00 9 2000-10-01 23:58:00 12 2000-10-02 00:05:00 15 2000-10-02 00:12:00 18 2000-10-02 00:19:00 21 2000-10-02 00:26:00 24 Freq: 7T, dtype: int64
>>> ts.resample('17min').sum() 2000-10-01 23:14:00 0 2000-10-01 23:31:00 9 2000-10-01 23:48:00 21 2000-10-02 00:05:00 54 2000-10-02 00:22:00 24 Freq: 17T, dtype: int64
>>> ts.resample('17min', origin='epoch').sum() 2000-10-01 23:18:00 0 2000-10-01 23:35:00 18 2000-10-01 23:52:00 27 2000-10-02 00:09:00 39 2000-10-02 00:26:00 24 Freq: 17T, dtype: int64
>>> ts.resample('17min', origin='2000-01-01').sum() 2000-10-01 23:24:00 3 2000-10-01 23:41:00 15 2000-10-01 23:58:00 45 2000-10-02 00:15:00 45 Freq: 17T, dtype: int64
If you want to adjust the start of the bins with an offset Timedelta, the two following lines are equivalent:
>>> ts.resample('17min', origin='start').sum() 2000-10-01 23:30:00 9 2000-10-01 23:47:00 21 2000-10-02 00:04:00 54 2000-10-02 00:21:00 24 Freq: 17T, dtype: int64
>>> ts.resample('17min', offset='23h30min').sum() 2000-10-01 23:30:00 9 2000-10-01 23:47:00 21 2000-10-02 00:04:00 54 2000-10-02 00:21:00 24 Freq: 17T, dtype: int64
If you want to take the largest Timestamp as the end of the bins:
>>> ts.resample('17min', origin='end').sum() 2000-10-01 23:35:00 0 2000-10-01 23:52:00 18 2000-10-02 00:09:00 27 2000-10-02 00:26:00 63 Freq: 17T, dtype: int64
In contrast with the start_day, you can use end_day to take the ceiling midnight of the largest Timestamp as the end of the bins and drop the bins not containing data:
>>> ts.resample('17min', origin='end_day').sum() 2000-10-01 23:38:00 3 2000-10-01 23:55:00 15 2000-10-02 00:12:00 45 2000-10-02 00:29:00 45 Freq: 17T, dtype: int64
To replace the use of the deprecated base argument, you can now use offset, in this example it is equivalent to have base=2:
>>> ts.resample('17min', offset='2min').sum() 2000-10-01 23:16:00 0 2000-10-01 23:33:00 9 2000-10-01 23:50:00 36 2000-10-02 00:07:00 39 2000-10-02 00:24:00 24 Freq: 17T, dtype: int64
To replace the use of the deprecated loffset argument:
>>> from pandas.tseries.frequencies import to_offset >>> loffset = '19min' >>> ts_out = ts.resample('17min').sum() >>> ts_out.index = ts_out.index + to_offset(loffset) >>> ts_out 2000-10-01 23:33:00 0 2000-10-01 23:50:00 9 2000-10-02 00:07:00 21 2000-10-02 00:24:00 54 2000-10-02 00:41:00 24 Freq: 17T, dtype: int64
- reset_index(level=None, drop=False, name=NoDefault.no_default, inplace=False)[source]¶
Generate a new DataFrame or Series with the index reset.
This is useful when the index needs to be treated as a column, or when the index is meaningless and needs to be reset to the default before another operation.
- Parameters
level (int, str, tuple, or list, default optional) – For a Series with a MultiIndex, only remove the specified levels from the index. Removes all levels by default.
drop (bool, default False) – Just reset the index, without inserting it as a column in the new DataFrame.
name (object, optional) – The name to use for the column containing the original Series values. Uses
self.name
by default. This argument is ignored when drop is True.inplace (bool, default False) – Modify the Series in place (do not create a new object).
- Returns
When drop is False (the default), a DataFrame is returned. The newly created columns will come first in the DataFrame, followed by the original Series values. When drop is True, a Series is returned. In either case, if
inplace=True
, no value is returned.- Return type
See also
DataFrame.reset_index
Analogous function for DataFrame.
Examples
>>> s = pd.Series([1, 2, 3, 4], name='foo', ... index=pd.Index(['a', 'b', 'c', 'd'], name='idx'))
Generate a DataFrame with default index.
>>> s.reset_index() idx foo 0 a 1 1 b 2 2 c 3 3 d 4
To specify the name of the new column use name.
>>> s.reset_index(name='values') idx values 0 a 1 1 b 2 2 c 3 3 d 4
To generate a new Series with the default set drop to True.
>>> s.reset_index(drop=True) 0 1 1 2 2 3 3 4 Name: foo, dtype: int64
To update the Series in place, without generating a new one set inplace to True. Note that it also requires
drop=True
.>>> s.reset_index(inplace=True, drop=True) >>> s 0 1 1 2 2 3 3 4 Name: foo, dtype: int64
The level parameter is interesting for Series with a multi-level index.
>>> arrays = [np.array(['bar', 'bar', 'baz', 'baz']), ... np.array(['one', 'two', 'one', 'two'])] >>> s2 = pd.Series( ... range(4), name='foo', ... index=pd.MultiIndex.from_arrays(arrays, ... names=['a', 'b']))
To remove a specific level from the Index, use level.
>>> s2.reset_index(level='a') a foo b one bar 0 two bar 1 one baz 2 two baz 3
If level is not set, all levels are removed from the Index.
>>> s2.reset_index() a b foo 0 bar one 0 1 bar two 1 2 baz one 2 3 baz two 3
- rfloordiv(other, level=None, fill_value=None, axis=0)¶
Return Integer division of series and other, element-wise (binary operator rfloordiv).
Equivalent to
other // series
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.floordiv
Element-wise Integer division, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.floordiv(b, fill_value=0) a 1.0 b NaN c NaN d 0.0 e NaN dtype: float64
- rmod(other, level=None, fill_value=None, axis=0)¶
Return Modulo of series and other, element-wise (binary operator rmod).
Equivalent to
other % series
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.mod
Element-wise Modulo, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.mod(b, fill_value=0) a 0.0 b NaN c NaN d 0.0 e NaN dtype: float64
- rmul(other, level=None, fill_value=None, axis=0)¶
Return Multiplication of series and other, element-wise (binary operator rmul).
Equivalent to
other * series
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.mul
Element-wise Multiplication, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.multiply(b, fill_value=0) a 1.0 b 0.0 c 0.0 d 0.0 e NaN dtype: float64
- round(decimals=0, *args, **kwargs)[source]¶
Round each value in a Series to the given number of decimals.
- Parameters
decimals (int, default 0) – Number of decimal places to round to. If decimals is negative, it specifies the number of positions to the left of the decimal point.
*args, **kwargs – Additional arguments and keywords have no effect but might be accepted for compatibility with NumPy.
- Returns
Rounded values of the Series.
- Return type
See also
numpy.around
Round values of an np.array.
DataFrame.round
Round values of a DataFrame.
Examples
>>> s = pd.Series([0.1, 1.3, 2.7]) >>> s.round() 0 0.0 1 1.0 2 3.0 dtype: float64
- rpow(other, level=None, fill_value=None, axis=0)¶
Return Exponential power of series and other, element-wise (binary operator rpow).
Equivalent to
other ** series
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.pow
Element-wise Exponential power, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.pow(b, fill_value=0) a 1.0 b 1.0 c 1.0 d 0.0 e NaN dtype: float64
- rsub(other, level=None, fill_value=None, axis=0)¶
Return Subtraction of series and other, element-wise (binary operator rsub).
Equivalent to
other - series
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.sub
Element-wise Subtraction, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.subtract(b, fill_value=0) a 0.0 b 1.0 c 1.0 d -1.0 e NaN dtype: float64
- rtruediv(other, level=None, fill_value=None, axis=0)¶
Return Floating division of series and other, element-wise (binary operator rtruediv).
Equivalent to
other / series
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.truediv
Element-wise Floating division, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.divide(b, fill_value=0) a 1.0 b inf c inf d 0.0 e NaN dtype: float64
- searchsorted(value, side='left', sorter=None)[source]¶
Find indices where elements should be inserted to maintain order.
Find the indices into a sorted Series self such that, if the corresponding elements in value were inserted before the indices, the order of self would be preserved.
Note
The Series must be monotonically sorted, otherwise wrong locations will likely be returned. Pandas does not check this for you.
- Parameters
value (array-like or scalar) – Values to insert into self.
side ({‘left’, ‘right’}, optional) – If ‘left’, the index of the first suitable location found is given. If ‘right’, return the last such index. If there is no suitable index, return either 0 or N (where N is the length of self).
sorter (1-D array-like, optional) – Optional array of integer indices that sort self into ascending order. They are typically the result of
np.argsort
.
- Returns
A scalar or array of insertion points with the same shape as value.
- Return type
int or array of int
See also
sort_values
Sort by the values along either axis.
numpy.searchsorted
Similar method from NumPy.
Notes
Binary search is used to find the required insertion points.
Examples
>>> ser = pd.Series([1, 2, 3]) >>> ser 0 1 1 2 2 3 dtype: int64
>>> ser.searchsorted(4) 3
>>> ser.searchsorted([0, 4]) array([0, 3])
>>> ser.searchsorted([1, 3], side='left') array([0, 2])
>>> ser.searchsorted([1, 3], side='right') array([1, 3])
>>> ser = pd.Series(pd.to_datetime(['3/11/2000', '3/12/2000', '3/13/2000'])) >>> ser 0 2000-03-11 1 2000-03-12 2 2000-03-13 dtype: datetime64[ns]
>>> ser.searchsorted('3/14/2000') 3
>>> ser = pd.Categorical( ... ['apple', 'bread', 'bread', 'cheese', 'milk'], ordered=True ... ) >>> ser ['apple', 'bread', 'bread', 'cheese', 'milk'] Categories (4, object): ['apple' < 'bread' < 'cheese' < 'milk']
>>> ser.searchsorted('bread') 1
>>> ser.searchsorted(['bread'], side='right') array([3])
If the values are not monotonically sorted, wrong locations may be returned:
>>> ser = pd.Series([2, 1, 3]) >>> ser 0 2 1 1 2 3 dtype: int64
>>> ser.searchsorted(1) 0 # wrong result, correct would be 1
- sem(axis=None, skipna=True, level=None, ddof=1, numeric_only=None, **kwargs)¶
Return unbiased standard error of the mean over requested axis.
Normalized by N-1 by default. This can be changed using the ddof argument
- Parameters
axis ({index (0)})
skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar.
ddof (int, default 1) – Delta Degrees of Freedom. The divisor used in calculations is N - ddof, where N represents the number of elements.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
- Return type
scalar or Series (if level specified)
- set_axis(labels, axis: Axis = 0, inplace: Literal[False] = False) Series [source]¶
- set_axis(labels, axis: Axis, inplace: Literal[True]) None
- set_axis(labels, *, inplace: Literal[True]) None
- set_axis(labels, axis: Axis = 0, inplace: bool = False) Series | None
Assign desired index to given axis.
Indexes for row labels can be changed by assigning a list-like or Index.
- Parameters
labels (list-like, Index) – The values for the new index.
axis ({0 or ‘index’}, default 0) – The axis to update. The value 0 identifies the rows.
inplace (bool, default False) – Whether to return a new Series instance.
- Returns
renamed – An object of type Series or None if
inplace=True
.- Return type
Series or None
- shift(periods=1, freq=None, axis=0, fill_value=None)[source]¶
Shift index by desired number of periods with an optional time freq.
When freq is not passed, shift the index without realigning the data. If freq is passed (in this case, the index must be date or datetime, or it will raise a NotImplementedError), the index will be increased using the periods and the freq. freq can be inferred when specified as “infer” as long as either freq or inferred_freq attribute is set in the index.
- Parameters
periods (int) – Number of periods to shift. Can be positive or negative.
freq (DateOffset, tseries.offsets, timedelta, or str, optional) – Offset to use from the tseries module or time rule (e.g. ‘EOM’). If freq is specified then the index values are shifted but the data is not realigned. That is, use freq if you would like to extend the index when shifting and preserve the original data. If freq is specified as “infer” then it will be inferred from the freq or inferred_freq attributes of the index. If neither of those attributes exist, a ValueError is thrown.
axis ({0 or ‘index’, 1 or ‘columns’, None}, default None) – Shift direction.
fill_value (object, optional) – The scalar value to use for newly introduced missing values. the default depends on the dtype of self. For numeric data,
np.nan
is used. For datetime, timedelta, or period data, etc.NaT
is used. For extension dtypes,self.dtype.na_value
is used.Changed in version 1.1.0.
- Returns
Copy of input object, shifted.
- Return type
See also
Index.shift
Shift values of Index.
DatetimeIndex.shift
Shift values of DatetimeIndex.
PeriodIndex.shift
Shift values of PeriodIndex.
tshift
Shift the time index, using the index’s frequency if available.
Examples
>>> df = pd.DataFrame({"Col1": [10, 20, 15, 30, 45], ... "Col2": [13, 23, 18, 33, 48], ... "Col3": [17, 27, 22, 37, 52]}, ... index=pd.date_range("2020-01-01", "2020-01-05")) >>> df Col1 Col2 Col3 2020-01-01 10 13 17 2020-01-02 20 23 27 2020-01-03 15 18 22 2020-01-04 30 33 37 2020-01-05 45 48 52
>>> df.shift(periods=3) Col1 Col2 Col3 2020-01-01 NaN NaN NaN 2020-01-02 NaN NaN NaN 2020-01-03 NaN NaN NaN 2020-01-04 10.0 13.0 17.0 2020-01-05 20.0 23.0 27.0
>>> df.shift(periods=1, axis="columns") Col1 Col2 Col3 2020-01-01 NaN 10 13 2020-01-02 NaN 20 23 2020-01-03 NaN 15 18 2020-01-04 NaN 30 33 2020-01-05 NaN 45 48
>>> df.shift(periods=3, fill_value=0) Col1 Col2 Col3 2020-01-01 0 0 0 2020-01-02 0 0 0 2020-01-03 0 0 0 2020-01-04 10 13 17 2020-01-05 20 23 27
>>> df.shift(periods=3, freq="D") Col1 Col2 Col3 2020-01-04 10 13 17 2020-01-05 20 23 27 2020-01-06 15 18 22 2020-01-07 30 33 37 2020-01-08 45 48 52
>>> df.shift(periods=3, freq="infer") Col1 Col2 Col3 2020-01-04 10 13 17 2020-01-05 20 23 27 2020-01-06 15 18 22 2020-01-07 30 33 37 2020-01-08 45 48 52
- skew(axis: int | None | lib.NoDefault = NoDefault.no_default, skipna=True, level=None, numeric_only=None, **kwargs)¶
Return unbiased skew over requested axis.
Normalized by N-1.
- Parameters
axis ({index (0)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
**kwargs – Additional keyword arguments to be passed to the function.
- Return type
scalar or Series (if level specified)
- sort_index(axis=0, level=None, ascending=True, inplace=False, kind='quicksort', na_position='last', sort_remaining=True, ignore_index=False, key=None)[source]¶
Sort Series by index labels.
Returns a new Series sorted by label if inplace argument is
False
, otherwise updates the original series and returns None.- Parameters
axis (int, default 0) – Axis to direct sorting. This can only be 0 for Series.
level (int, optional) – If not None, sort on values in specified index level(s).
ascending (bool or list-like of bools, default True) – Sort ascending vs. descending. When the index is a MultiIndex the sort direction can be controlled for each level individually.
inplace (bool, default False) – If True, perform operation in-place.
kind ({‘quicksort’, ‘mergesort’, ‘heapsort’, ‘stable’}, default ‘quicksort’) – Choice of sorting algorithm. See also
numpy.sort()
for more information. ‘mergesort’ and ‘stable’ are the only stable algorithms. For DataFrames, this option is only applied when sorting on a single column or label.na_position ({‘first’, ‘last’}, default ‘last’) – If ‘first’ puts NaNs at the beginning, ‘last’ puts NaNs at the end. Not implemented for MultiIndex.
sort_remaining (bool, default True) – If True and sorting by level and index is multilevel, sort by other levels too (in order) after sorting by specified level.
ignore_index (bool, default False) – If True, the resulting axis will be labeled 0, 1, …, n - 1.
New in version 1.0.0.
key (callable, optional) – If not None, apply the key function to the index values before sorting. This is similar to the key argument in the builtin
sorted()
function, with the notable difference that this key function should be vectorized. It should expect anIndex
and return anIndex
of the same shape.New in version 1.1.0.
- Returns
The original Series sorted by the labels or None if
inplace=True
.- Return type
Series or None
See also
DataFrame.sort_index
Sort DataFrame by the index.
DataFrame.sort_values
Sort DataFrame by the value.
Series.sort_values
Sort Series by the value.
Examples
>>> s = pd.Series(['a', 'b', 'c', 'd'], index=[3, 2, 1, 4]) >>> s.sort_index() 1 c 2 b 3 a 4 d dtype: object
Sort Descending
>>> s.sort_index(ascending=False) 4 d 3 a 2 b 1 c dtype: object
Sort Inplace
>>> s.sort_index(inplace=True) >>> s 1 c 2 b 3 a 4 d dtype: object
By default NaNs are put at the end, but use na_position to place them at the beginning
>>> s = pd.Series(['a', 'b', 'c', 'd'], index=[3, 2, 1, np.nan]) >>> s.sort_index(na_position='first') NaN d 1.0 c 2.0 b 3.0 a dtype: object
Specify index level to sort
>>> arrays = [np.array(['qux', 'qux', 'foo', 'foo', ... 'baz', 'baz', 'bar', 'bar']), ... np.array(['two', 'one', 'two', 'one', ... 'two', 'one', 'two', 'one'])] >>> s = pd.Series([1, 2, 3, 4, 5, 6, 7, 8], index=arrays) >>> s.sort_index(level=1) bar one 8 baz one 6 foo one 4 qux one 2 bar two 7 baz two 5 foo two 3 qux two 1 dtype: int64
Does not sort by remaining levels when sorting by levels
>>> s.sort_index(level=1, sort_remaining=False) qux one 2 foo one 4 baz one 6 bar one 8 qux two 1 foo two 3 baz two 5 bar two 7 dtype: int64
Apply a key function before sorting
>>> s = pd.Series([1, 2, 3, 4], index=['A', 'b', 'C', 'd']) >>> s.sort_index(key=lambda x : x.str.lower()) A 1 b 2 C 3 d 4 dtype: int64
- sort_values(axis=0, ascending=True, inplace=False, kind='quicksort', na_position='last', ignore_index=False, key=None)[source]¶
Sort by the values.
Sort a Series in ascending or descending order by some criterion.
- Parameters
axis ({0 or ‘index’}, default 0) – Axis to direct sorting. The value ‘index’ is accepted for compatibility with DataFrame.sort_values.
ascending (bool or list of bools, default True) – If True, sort values in ascending order, otherwise descending.
inplace (bool, default False) – If True, perform operation in-place.
kind ({‘quicksort’, ‘mergesort’, ‘heapsort’, ‘stable’}, default ‘quicksort’) – Choice of sorting algorithm. See also
numpy.sort()
for more information. ‘mergesort’ and ‘stable’ are the only stable algorithms.na_position ({‘first’ or ‘last’}, default ‘last’) – Argument ‘first’ puts NaNs at the beginning, ‘last’ puts NaNs at the end.
ignore_index (bool, default False) – If True, the resulting axis will be labeled 0, 1, …, n - 1.
New in version 1.0.0.
key (callable, optional) – If not None, apply the key function to the series values before sorting. This is similar to the key argument in the builtin
sorted()
function, with the notable difference that this key function should be vectorized. It should expect aSeries
and return an array-like.New in version 1.1.0.
- Returns
Series ordered by values or None if
inplace=True
.- Return type
Series or None
See also
Series.sort_index
Sort by the Series indices.
DataFrame.sort_values
Sort DataFrame by the values along either axis.
DataFrame.sort_index
Sort DataFrame by indices.
Examples
>>> s = pd.Series([np.nan, 1, 3, 10, 5]) >>> s 0 NaN 1 1.0 2 3.0 3 10.0 4 5.0 dtype: float64
Sort values ascending order (default behaviour)
>>> s.sort_values(ascending=True) 1 1.0 2 3.0 4 5.0 3 10.0 0 NaN dtype: float64
Sort values descending order
>>> s.sort_values(ascending=False) 3 10.0 4 5.0 2 3.0 1 1.0 0 NaN dtype: float64
Sort values inplace
>>> s.sort_values(ascending=False, inplace=True) >>> s 3 10.0 4 5.0 2 3.0 1 1.0 0 NaN dtype: float64
Sort values putting NAs first
>>> s.sort_values(na_position='first') 0 NaN 1 1.0 2 3.0 4 5.0 3 10.0 dtype: float64
Sort a series of strings
>>> s = pd.Series(['z', 'b', 'd', 'a', 'c']) >>> s 0 z 1 b 2 d 3 a 4 c dtype: object
>>> s.sort_values() 3 a 1 b 4 c 2 d 0 z dtype: object
Sort using a key function. Your key function will be given the
Series
of values and should return an array-like.>>> s = pd.Series(['a', 'B', 'c', 'D', 'e']) >>> s.sort_values() 1 B 3 D 0 a 2 c 4 e dtype: object >>> s.sort_values(key=lambda x: x.str.lower()) 0 a 1 B 2 c 3 D 4 e dtype: object
NumPy ufuncs work well here. For example, we can sort by the
sin
of the value>>> s = pd.Series([-4, -2, 0, 2, 4]) >>> s.sort_values(key=np.sin) 1 -2 4 4 2 0 0 -4 3 2 dtype: int64
More complicated user-defined functions can be used, as long as they expect a Series and return an array-like
>>> s.sort_values(key=lambda x: (np.tan(x.cumsum()))) 0 -4 3 2 4 4 1 -2 2 0 dtype: int64
- sparse¶
alias of
pandas.core.arrays.sparse.accessor.SparseAccessor
- std(axis=None, skipna=True, level=None, ddof=1, numeric_only=None, **kwargs)¶
Return sample standard deviation over requested axis.
Normalized by N-1 by default. This can be changed using the ddof argument.
- Parameters
axis ({index (0)})
skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar.
ddof (int, default 1) – Delta Degrees of Freedom. The divisor used in calculations is N - ddof, where N represents the number of elements.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
- Return type
scalar or Series (if level specified)
Notes
To have the same behaviour as numpy.std, use ddof=0 (instead of the default ddof=1)
Examples
>>> df = pd.DataFrame({'person_id': [0, 1, 2, 3], ... 'age': [21, 25, 62, 43], ... 'height': [1.61, 1.87, 1.49, 2.01]} ... ).set_index('person_id') >>> df age height person_id 0 21 1.61 1 25 1.87 2 62 1.49 3 43 2.01
The standard deviation of the columns can be found as follows:
>>> df.std() age 18.786076 height 0.237417
Alternatively, ddof=0 can be set to normalize by N instead of N-1:
>>> df.std(ddof=0) age 16.269219 height 0.205609
- str¶
alias of
pandas.core.strings.accessor.StringMethods
- sub(other, level=None, fill_value=None, axis=0)¶
Return Subtraction of series and other, element-wise (binary operator sub).
Equivalent to
series - other
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.rsub
Reverse of the Subtraction operator, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.subtract(b, fill_value=0) a 0.0 b 1.0 c 1.0 d -1.0 e NaN dtype: float64
- subtract(other, level=None, fill_value=None, axis=0)¶
Return Subtraction of series and other, element-wise (binary operator sub).
Equivalent to
series - other
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.rsub
Reverse of the Subtraction operator, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.subtract(b, fill_value=0) a 0.0 b 1.0 c 1.0 d -1.0 e NaN dtype: float64
- sum(axis=None, skipna=True, level=None, numeric_only=None, min_count=0, **kwargs)¶
Return the sum of the values over the requested axis.
This is equivalent to the method
numpy.sum
.- Parameters
axis ({index (0)}) – Axis for the function to be applied on.
skipna (bool, default True) – Exclude NA/null values when computing the result.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
min_count (int, default 0) – The required number of valid values to perform the operation. If fewer than
min_count
non-NA values are present the result will be NA.**kwargs – Additional keyword arguments to be passed to the function.
- Return type
scalar or Series (if level specified)
See also
Series.sum
Return the sum.
Series.min
Return the minimum.
Series.max
Return the maximum.
Series.idxmin
Return the index of the minimum.
Series.idxmax
Return the index of the maximum.
DataFrame.sum
Return the sum over the requested axis.
DataFrame.min
Return the minimum over the requested axis.
DataFrame.max
Return the maximum over the requested axis.
DataFrame.idxmin
Return the index of the minimum over the requested axis.
DataFrame.idxmax
Return the index of the maximum over the requested axis.
Examples
>>> idx = pd.MultiIndex.from_arrays([ ... ['warm', 'warm', 'cold', 'cold'], ... ['dog', 'falcon', 'fish', 'spider']], ... names=['blooded', 'animal']) >>> s = pd.Series([4, 2, 0, 8], name='legs', index=idx) >>> s blooded animal warm dog 4 falcon 2 cold fish 0 spider 8 Name: legs, dtype: int64
>>> s.sum() 14
By default, the sum of an empty or all-NA Series is
0
.>>> pd.Series([], dtype="float64").sum() # min_count=0 is the default 0.0
This can be controlled with the
min_count
parameter. For example, if you’d like the sum of an empty series to be NaN, passmin_count=1
.>>> pd.Series([], dtype="float64").sum(min_count=1) nan
Thanks to the
skipna
parameter,min_count
handles all-NA and empty series identically.>>> pd.Series([np.nan]).sum() 0.0
>>> pd.Series([np.nan]).sum(min_count=1) nan
- swaplevel(i=- 2, j=- 1, copy=True)[source]¶
Swap levels i and j in a
MultiIndex
.Default is to swap the two innermost levels of the index.
- Parameters
i, j (int or str) – Levels of the indices to be swapped. Can pass level name as string.
copy (bool, default True) – Whether to copy underlying data.
- Returns
Series with levels swapped in MultiIndex.
- Return type
Examples
>>> s = pd.Series( ... ["A", "B", "A", "C"], ... index=[ ... ["Final exam", "Final exam", "Coursework", "Coursework"], ... ["History", "Geography", "History", "Geography"], ... ["January", "February", "March", "April"], ... ], ... ) >>> s Final exam History January A Geography February B Coursework History March A Geography April C dtype: object
In the following example, we will swap the levels of the indices. Here, we will swap the levels column-wise, but levels can be swapped row-wise in a similar manner. Note that column-wise is the default behaviour. By not supplying any arguments for i and j, we swap the last and second to last indices.
>>> s.swaplevel() Final exam January History A February Geography B Coursework March History A April Geography C dtype: object
By supplying one argument, we can choose which index to swap the last index with. We can for example swap the first index with the last one as follows.
>>> s.swaplevel(0) January History Final exam A February Geography Final exam B March History Coursework A April Geography Coursework C dtype: object
We can also define explicitly which indices we want to swap by supplying values for both i and j. Here, we for example swap the first and second indices.
>>> s.swaplevel(0, 1) History Final exam January A Geography Final exam February B History Coursework March A Geography Coursework April C dtype: object
- take(indices, axis=0, is_copy=None, **kwargs)[source]¶
Return the elements in the given positional indices along an axis.
This means that we are not indexing according to actual values in the index attribute of the object. We are indexing according to the actual position of the element in the object.
- Parameters
indices (array-like) – An array of ints indicating which positions to take.
axis ({0 or ‘index’, 1 or ‘columns’, None}, default 0) – The axis on which to select elements.
0
means that we are selecting rows,1
means that we are selecting columns.is_copy (bool) – Before pandas 1.0,
is_copy=False
can be specified to ensure that the return value is an actual copy. Starting with pandas 1.0,take
always returns a copy, and the keyword is therefore deprecated.Deprecated since version 1.0.0.
**kwargs – For compatibility with
numpy.take()
. Has no effect on the output.
- Returns
taken – An array-like containing the elements taken from the object.
- Return type
same type as caller
See also
DataFrame.loc
Select a subset of a DataFrame by labels.
DataFrame.iloc
Select a subset of a DataFrame by positions.
numpy.take
Take elements from an array along an axis.
Examples
>>> df = pd.DataFrame([('falcon', 'bird', 389.0), ... ('parrot', 'bird', 24.0), ... ('lion', 'mammal', 80.5), ... ('monkey', 'mammal', np.nan)], ... columns=['name', 'class', 'max_speed'], ... index=[0, 2, 3, 1]) >>> df name class max_speed 0 falcon bird 389.0 2 parrot bird 24.0 3 lion mammal 80.5 1 monkey mammal NaN
Take elements at positions 0 and 3 along the axis 0 (default).
Note how the actual indices selected (0 and 1) do not correspond to our selected indices 0 and 3. That’s because we are selecting the 0th and 3rd rows, not rows whose indices equal 0 and 3.
>>> df.take([0, 3]) name class max_speed 0 falcon bird 389.0 1 monkey mammal NaN
Take elements at indices 1 and 2 along the axis 1 (column selection).
>>> df.take([1, 2], axis=1) class max_speed 0 bird 389.0 2 bird 24.0 3 mammal 80.5 1 mammal NaN
We may take elements using negative integers for positive indices, starting from the end of the object, just like with Python lists.
>>> df.take([-1, -2]) name class max_speed 1 monkey mammal NaN 3 lion mammal 80.5
- to_dict(into=<class 'dict'>)[source]¶
Convert Series to {label -> value} dict or dict-like object.
- Parameters
into (class, default dict) – The collections.abc.Mapping subclass to use as the return object. Can be the actual class or an empty instance of the mapping type you want. If you want a collections.defaultdict, you must pass it initialized.
- Returns
Key-value representation of Series.
- Return type
Examples
>>> s = pd.Series([1, 2, 3, 4]) >>> s.to_dict() {0: 1, 1: 2, 2: 3, 3: 4} >>> from collections import OrderedDict, defaultdict >>> s.to_dict(OrderedDict) OrderedDict([(0, 1), (1, 2), (2, 3), (3, 4)]) >>> dd = defaultdict(list) >>> s.to_dict(dd) defaultdict(<class 'list'>, {0: 1, 1: 2, 2: 3, 3: 4})
- to_frame(name=NoDefault.no_default)[source]¶
Convert Series to DataFrame.
- Parameters
name (object, optional) – The passed name should substitute for the series name (if it has one).
- Returns
DataFrame representation of Series.
- Return type
Examples
>>> s = pd.Series(["a", "b", "c"], ... name="vals") >>> s.to_frame() vals 0 a 1 b 2 c
- to_markdown(buf=None, mode='wt', index=True, storage_options=None, **kwargs)[source]¶
Print Series in Markdown-friendly format.
New in version 1.0.0.
- Parameters
buf (str, Path or StringIO-like, optional, default None) – Buffer to write to. If None, the output is returned as a string.
mode (str, optional) – Mode in which file is opened, “wt” by default.
index (bool, optional, default True) – Add index (row) labels.
New in version 1.1.0.
storage_options (dict, optional) – Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc. For HTTP(S) URLs the key-value pairs are forwarded to
urllib
as header options. For other URLs (e.g. starting with “s3://”, and “gcs://”) the key-value pairs are forwarded tofsspec
. Please seefsspec
andurllib
for more details.New in version 1.2.0.
**kwargs – These parameters will be passed to tabulate.
- Returns
Series in Markdown-friendly format.
- Return type
Notes
Requires the tabulate package.
- Examples
>>> s = pd.Series(["elk", "pig", "dog", "quetzal"], name="animal") >>> print(s.to_markdown()) | | animal | |---:|:---------| | 0 | elk | | 1 | pig | | 2 | dog | | 3 | quetzal |
Output markdown with a tabulate option.
>>> print(s.to_markdown(tablefmt="grid")) +----+----------+ | | animal | +====+==========+ | 0 | elk | +----+----------+ | 1 | pig | +----+----------+ | 2 | dog | +----+----------+ | 3 | quetzal | +----+----------+
- to_period(freq=None, copy=True)[source]¶
Convert Series from DatetimeIndex to PeriodIndex.
- Parameters
freq (str, default None) – Frequency associated with the PeriodIndex.
copy (bool, default True) – Whether or not to return a copy.
- Returns
Series with index converted to PeriodIndex.
- Return type
- to_string(buf=None, na_rep='NaN', float_format=None, header=True, index=True, length=False, dtype=False, name=False, max_rows=None, min_rows=None)[source]¶
Render a string representation of the Series.
- Parameters
buf (StringIO-like, optional) – Buffer to write to.
na_rep (str, optional) – String representation of NaN to use, default ‘NaN’.
float_format (one-parameter function, optional) – Formatter function to apply to columns’ elements if they are floats, default None.
header (bool, default True) – Add the Series header (index name).
index (bool, optional) – Add index (row) labels, default True.
length (bool, default False) – Add the Series length.
dtype (bool, default False) – Add the Series dtype.
name (bool, default False) – Add the Series name if not None.
max_rows (int, optional) – Maximum number of rows to show before truncating. If None, show all.
min_rows (int, optional) – The number of rows to display in a truncated repr (when number of rows is above max_rows).
- Returns
String representation of Series if
buf=None
, otherwise None.- Return type
str or None
- to_timestamp(freq=None, how='start', copy=True)[source]¶
Cast to DatetimeIndex of Timestamps, at beginning of period.
- Parameters
freq (str, default frequency of PeriodIndex) – Desired frequency.
how ({‘s’, ‘e’, ‘start’, ‘end’}) – Convention for converting period to timestamp; start of period vs. end.
copy (bool, default True) – Whether or not to return a copy.
- Return type
Series with DatetimeIndex
- transform(func, axis=0, *args, **kwargs)[source]¶
Call
func
on self producing a Series with the same axis shape as self.- Parameters
func (function, str, list-like or dict-like) – Function to use for transforming the data. If a function, must either work when passed a Series or when passed to Series.apply. If func is both list-like and dict-like, dict-like behavior takes precedence.
Accepted combinations are:
function
string function name
list-like of functions and/or function names, e.g.
[np.exp, 'sqrt']
dict-like of axis labels -> functions, function names or list-like of such.
axis ({0 or ‘index’}) – Parameter needed for compatibility with DataFrame.
*args – Positional arguments to pass to func.
**kwargs – Keyword arguments to pass to func.
- Returns
A Series that must have the same length as self.
- Return type
:raises ValueError : If the returned Series has a different length than self.:
See also
Series.agg
Only perform aggregating type operations.
Series.apply
Invoke function on a Series.
Notes
Functions that mutate the passed object can produce unexpected behavior or errors and are not supported. See Mutating with User Defined Function (UDF) methods for more details.
Examples
>>> df = pd.DataFrame({'A': range(3), 'B': range(1, 4)}) >>> df A B 0 0 1 1 1 2 2 2 3 >>> df.transform(lambda x: x + 1) A B 0 1 2 1 2 3 2 3 4
Even though the resulting Series must have the same length as the input Series, it is possible to provide several input functions:
>>> s = pd.Series(range(3)) >>> s 0 0 1 1 2 2 dtype: int64 >>> s.transform([np.sqrt, np.exp]) sqrt exp 0 0.000000 1.000000 1 1.000000 2.718282 2 1.414214 7.389056
You can call transform on a GroupBy object:
>>> df = pd.DataFrame({ ... "Date": [ ... "2015-05-08", "2015-05-07", "2015-05-06", "2015-05-05", ... "2015-05-08", "2015-05-07", "2015-05-06", "2015-05-05"], ... "Data": [5, 8, 6, 1, 50, 100, 60, 120], ... }) >>> df Date Data 0 2015-05-08 5 1 2015-05-07 8 2 2015-05-06 6 3 2015-05-05 1 4 2015-05-08 50 5 2015-05-07 100 6 2015-05-06 60 7 2015-05-05 120 >>> df.groupby('Date')['Data'].transform('sum') 0 55 1 108 2 66 3 121 4 55 5 108 6 66 7 121 Name: Data, dtype: int64
>>> df = pd.DataFrame({ ... "c": [1, 1, 1, 2, 2, 2, 2], ... "type": ["m", "n", "o", "m", "m", "n", "n"] ... }) >>> df c type 0 1 m 1 1 n 2 1 o 3 2 m 4 2 m 5 2 n 6 2 n >>> df['size'] = df.groupby('c')['type'].transform(len) >>> df c type size 0 1 m 3 1 1 n 3 2 1 o 3 3 2 m 4 4 2 m 4 5 2 n 4 6 2 n 4
- truediv(other, level=None, fill_value=None, axis=0)¶
Return Floating division of series and other, element-wise (binary operator truediv).
Equivalent to
series / other
, but with support to substitute a fill_value for missing data in either one of the inputs.- Parameters
other (Series or scalar value)
fill_value (None or float value, default None (NaN)) – Fill existing missing (NaN) values, and any new element needed for successful Series alignment, with this value before computation. If data in both corresponding Series locations is missing the result of filling (at that location) will be missing.
level (int or name) – Broadcast across a level, matching Index values on the passed MultiIndex level.
- Returns
The result of the operation.
- Return type
See also
Series.rtruediv
Reverse of the Floating division operator, see Python documentation for more details.
Examples
>>> a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd']) >>> a a 1.0 b 1.0 c 1.0 d NaN dtype: float64 >>> b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e']) >>> b a 1.0 b NaN d 1.0 e NaN dtype: float64 >>> a.divide(b, fill_value=0) a 1.0 b inf c inf d 0.0 e NaN dtype: float64
- unique()[source]¶
Return unique values of Series object.
Uniques are returned in order of appearance. Hash table-based unique, therefore does NOT sort.
- Returns
The unique values returned as a NumPy array. See Notes.
- Return type
ndarray or ExtensionArray
See also
unique
Top-level unique method for any 1-d array-like object.
Index.unique
Return Index with unique values from an Index object.
Notes
Returns the unique values as a NumPy array. In case of an extension-array backed Series, a new
ExtensionArray
of that type with just the unique values is returned. This includesCategorical
Period
Datetime with Timezone
Interval
Sparse
IntegerNA
See Examples section.
Examples
>>> pd.Series([2, 1, 3, 3], name='A').unique() array([2, 1, 3])
>>> pd.Series([pd.Timestamp('2016-01-01') for _ in range(3)]).unique() array(['2016-01-01T00:00:00.000000000'], dtype='datetime64[ns]')
>>> pd.Series([pd.Timestamp('2016-01-01', tz='US/Eastern') ... for _ in range(3)]).unique() <DatetimeArray> ['2016-01-01 00:00:00-05:00'] Length: 1, dtype: datetime64[ns, US/Eastern]
An Categorical will return categories in the order of appearance and with the same dtype.
>>> pd.Series(pd.Categorical(list('baabc'))).unique() ['b', 'a', 'c'] Categories (3, object): ['a', 'b', 'c'] >>> pd.Series(pd.Categorical(list('baabc'), categories=list('abc'), ... ordered=True)).unique() ['b', 'a', 'c'] Categories (3, object): ['a' < 'b' < 'c']
- unstack(level=- 1, fill_value=None)[source]¶
Unstack, also known as pivot, Series with MultiIndex to produce DataFrame.
- Parameters
level (int, str, or list of these, default last level) – Level(s) to unstack, can pass level name.
fill_value (scalar value, default None) – Value to use when replacing NaN values.
- Returns
Unstacked Series.
- Return type
Notes
Reference the user guide for more examples.
Examples
>>> s = pd.Series([1, 2, 3, 4], ... index=pd.MultiIndex.from_product([['one', 'two'], ... ['a', 'b']])) >>> s one a 1 b 2 two a 3 b 4 dtype: int64
>>> s.unstack(level=-1) a b one 1 2 two 3 4
>>> s.unstack(level=0) one two a 1 3 b 2 4
- update(other)[source]¶
Modify Series in place using values from passed Series.
Uses non-NA values from passed Series to make updates. Aligns on index.
- Parameters
other (Series, or object coercible into Series)
Examples
>>> s = pd.Series([1, 2, 3]) >>> s.update(pd.Series([4, 5, 6])) >>> s 0 4 1 5 2 6 dtype: int64
>>> s = pd.Series(['a', 'b', 'c']) >>> s.update(pd.Series(['d', 'e'], index=[0, 2])) >>> s 0 d 1 b 2 e dtype: object
>>> s = pd.Series([1, 2, 3]) >>> s.update(pd.Series([4, 5, 6, 7, 8])) >>> s 0 4 1 5 2 6 dtype: int64
If
other
contains NaNs the corresponding values are not updated in the original Series.>>> s = pd.Series([1, 2, 3]) >>> s.update(pd.Series([4, np.nan, 6])) >>> s 0 4 1 2 2 6 dtype: int64
other
can also be a non-Series object type that is coercible into a Series>>> s = pd.Series([1, 2, 3]) >>> s.update([4, np.nan, 6]) >>> s 0 4 1 2 2 6 dtype: int64
>>> s = pd.Series([1, 2, 3]) >>> s.update({1: 9}) >>> s 0 1 1 9 2 3 dtype: int64
- Return type
- property values¶
Return Series as ndarray or ndarray-like depending on the dtype.
Warning
We recommend using
Series.array
orSeries.to_numpy()
, depending on whether you need a reference to the underlying data or a NumPy array.- Return type
numpy.ndarray or ndarray-like
See also
Series.array
Reference to the underlying data.
Series.to_numpy
A NumPy array representing the underlying data.
Examples
>>> pd.Series([1, 2, 3]).values array([1, 2, 3])
>>> pd.Series(list('aabc')).values array(['a', 'a', 'b', 'c'], dtype=object)
>>> pd.Series(list('aabc')).astype('category').values ['a', 'a', 'b', 'c'] Categories (3, object): ['a', 'b', 'c']
Timezone aware datetime data is converted to UTC:
>>> pd.Series(pd.date_range('20130101', periods=3, ... tz='US/Eastern')).values array(['2013-01-01T05:00:00.000000000', '2013-01-02T05:00:00.000000000', '2013-01-03T05:00:00.000000000'], dtype='datetime64[ns]')
- var(axis=None, skipna=True, level=None, ddof=1, numeric_only=None, **kwargs)¶
Return unbiased variance over requested axis.
Normalized by N-1 by default. This can be changed using the ddof argument.
- Parameters
axis ({index (0)})
skipna (bool, default True) – Exclude NA/null values. If an entire row/column is NA, the result will be NA.
level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a scalar.
ddof (int, default 1) – Delta Degrees of Freedom. The divisor used in calculations is N - ddof, where N represents the number of elements.
numeric_only (bool, default None) – Include only float, int, boolean columns. If None, will attempt to use everything, then use only numeric data. Not implemented for Series.
- Return type
scalar or Series (if level specified)
Examples
>>> df = pd.DataFrame({'person_id': [0, 1, 2, 3], ... 'age': [21, 25, 62, 43], ... 'height': [1.61, 1.87, 1.49, 2.01]} ... ).set_index('person_id') >>> df age height person_id 0 21 1.61 1 25 1.87 2 62 1.49 3 43 2.01
>>> df.var() age 352.916667 height 0.056367
Alternatively,
ddof=0
can be set to normalize by N instead of N-1:>>> df.var(ddof=0) age 264.687500 height 0.042275
- view(dtype=None)[source]¶
Create a new view of the Series.
This function will return a new Series with a view of the same underlying values in memory, optionally reinterpreted with a new data type. The new data type must preserve the same size in bytes as to not cause index misalignment.
- Parameters
dtype (data type) – Data type object or one of their string representations.
- Returns
A new Series object as a view of the same data in memory.
- Return type
See also
numpy.ndarray.view
Equivalent numpy function to create a new view of the same data in memory.
Notes
Series are instantiated with
dtype=float64
by default. Whilenumpy.ndarray.view()
will return a view with the same data type as the original array,Series.view()
(without specified dtype) will try usingfloat64
and may fail if the original data type size in bytes is not the same.Examples
>>> s = pd.Series([-2, -1, 0, 1, 2], dtype='int8') >>> s 0 -2 1 -1 2 0 3 1 4 2 dtype: int8
The 8 bit signed integer representation of -1 is 0b11111111, but the same bytes represent 255 if read as an 8 bit unsigned integer:
>>> us = s.view('uint8') >>> us 0 254 1 255 2 0 3 1 4 2 dtype: uint8
The views share the same underlying values:
>>> us[0] = 128 >>> s 0 -128 1 -1 2 0 3 1 4 2 dtype: int8
- where(cond, other=NoDefault.no_default, inplace=False, axis=None, level=None, errors=NoDefault.no_default, try_cast=NoDefault.no_default)[source]¶
Replace values where the condition is False.
- Parameters
cond (bool Series/DataFrame, array-like, or callable) – Where cond is True, keep the original value. Where False, replace with corresponding value from other. If cond is callable, it is computed on the Series/DataFrame and should return boolean Series/DataFrame or array. The callable must not change input Series/DataFrame (though pandas doesn’t check it).
other (scalar, Series/DataFrame, or callable) – Entries where cond is False are replaced with corresponding value from other. If other is callable, it is computed on the Series/DataFrame and should return scalar or Series/DataFrame. The callable must not change input Series/DataFrame (though pandas doesn’t check it).
inplace (bool, default False) – Whether to perform the operation in place on the data.
axis (int, default None) – Alignment axis if needed.
level (int, default None) – Alignment level if needed.
errors (str, {‘raise’, ‘ignore’}, default ‘raise’) – Note that currently this parameter won’t affect the results and will always coerce to a suitable dtype.
‘raise’ : allow exceptions to be raised.
‘ignore’ : suppress exceptions. On error return original object.
try_cast (bool, default None) – Try to cast the result back to the input type (if possible).
Deprecated since version 1.3.0: Manually cast back if necessary.
- Return type
Same type as caller or None if
inplace=True
.
See also
DataFrame.mask()
Return an object of same shape as self.
Notes
The where method is an application of the if-then idiom. For each element in the calling DataFrame, if
cond
isTrue
the element is used; otherwise the corresponding element from the DataFrameother
is used.The signature for
DataFrame.where()
differs fromnumpy.where()
. Roughlydf1.where(m, df2)
is equivalent tonp.where(m, df1, df2)
.For further details and examples see the
where
documentation in indexing.Examples
>>> s = pd.Series(range(5)) >>> s.where(s > 0) 0 NaN 1 1.0 2 2.0 3 3.0 4 4.0 dtype: float64 >>> s.mask(s > 0) 0 0.0 1 NaN 2 NaN 3 NaN 4 NaN dtype: float64
>>> s.where(s > 1, 10) 0 10 1 10 2 2 3 3 4 4 dtype: int64 >>> s.mask(s > 1, 10) 0 0 1 1 2 10 3 10 4 10 dtype: int64
>>> df = pd.DataFrame(np.arange(10).reshape(-1, 2), columns=['A', 'B']) >>> df A B 0 0 1 1 2 3 2 4 5 3 6 7 4 8 9 >>> m = df % 3 == 0 >>> df.where(m, -df) A B 0 0 -1 1 -2 3 2 -4 -5 3 6 -7 4 -8 9 >>> df.where(m, -df) == np.where(m, df, -df) A B 0 True True 1 True True 2 True True 3 True True 4 True True >>> df.where(m, -df) == df.mask(~m, -df) A B 0 True True 1 True True 2 True True 3 True True 4 True True