A bitcoin transaction typically looks as follows.
For each transaction there are,
180 (input) + 34 (output) + 10 (extra) = 224 bytes
If input is compressed 180 - 32 = 148 bytes
You can find the formula here
In the simplest case, you'll have one input and two outputs (the recipient, and change). So in=1 and out=2. According to the linked formula, the transaction size will be about
180 + 2 * 34+10 = 258 bytes
Today you can see most transactions with 1-10 satoshi/Byte.
1 satoshi = 0.00000001 BTC
Lets take 10 satoshi/Byte then 258 * 10 = 2580 satoshi = 0.00002580 BTC
Even if we have more inputs and outputs, we can say the transaction fee should be typically 0.0001 BTC. And you shouldn't pay more than that. BTC/USD rates are often fluctuating. Let us consider the price of 1 BTC = 10,000 USD. In this case, we pay 1 USD as a fee. You can always set the free around this amount for your transaction to go through.
Now let us understand how the transaction fee is decided. And how it is connected to bitcoin mining.
The block size is limited to 1 MB. So a miner has to choose transactions to fit in 1 MB of size. Each transaction comes with its own transaction fee.
Don't confuse block mining fee with the block reward of 12.5 BTC.
Let us say a transaction pays 0.01 BTC for it to be included in the block. And another transaction pays 0.015 BTC. And they are 512 KB each. So the miner gets 0.01 + 0.015 = 0.025 BTC for mining this block.
Another case can be, 4 transactions pay 0.01 BTC each with the size of 256 KB (256 * 4 = 1MB), now the total fee for the block will go up to 0.04 BTC. So the miner will profit more by including these smaller 4 transaction to improve his mining fee.
Here we understand that,
The Transaction fee in BTC does not matter to the miner. satoshi/Byte matters to him
In the above example we can calculate,
Satoshi per Byte in this case will be
For the other example with 512KB transaction size and 0.015 BTC fee,
For the other example with 256KB transaction size and 0.01 BTC fee,
The miner uses this calculation before adding a transaction to the block.
If the miner successfully mines a block, he gets all the transaction fees for the transactions he included in his block. So naturally, he will pick the transactions with the highest transaction fee.
Your transaction will not be confirmed unless it is included in a block. If you pay lower satoshi/Byte than what most people are paying, your transaction will not be picked up from the pool till better-paying transactions are there for miner.
Organizing transactions from an unconfirmed transaction pool into a block(1MB) with a maximum fee, makes the mining fee go up.
The miners don't pick up transactions with less fees. And those transactions are picked when higher fee transactions are not available in the unconfirmed pool. That is why we see delays in days for such transactions to confirm.
It is said a group of people wanted the bitcoin to become useless. So they inflated the transaction fee. How is that possible. They simply paid a large fee for the transaction.
So let us say someone pays 0.1 BTC for the transaction of size 256 KB. So fee becomes 39 satoshi/Byte. But still, this individual is letting in 4 other such transactions by paying 3.9 satoshi/Byte. So he adds a lot of unnecessary signatures to the input, making the size go up to 1 MB. Still paying a higher fee than others
(0.1/1024)* 10^5 = 9.7 satoshi/Byte and filling up all the space in the block.